The DOI matches its Crossref record; 39 of the 39 references it cites were found in a registry or an index.
The Option-Critic Architecture
Open accessAbstract
Temporal abstraction is key to scaling up learning and planning in reinforcement learning. While planning with temporally extended actions is well understood, creating such abstractions autonomously from data has remained challenging.We tackle this problem in the framework of options [Sutton,Precup and Singh, 1999; Precup, 2000]. We derive policy gradient theorems for options and propose a new option-critic architecture capable of learning both the internal policies and the termination conditions of options, in tandem with the policy over options, and without the need to provide any additional rewards or subgoals. Experimental results in both discrete and continuous environments showcase the flexibility and efficiency of the framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
(1999) · Cited by 5,147
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Deterministic Policy Gradient Algorithms
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., & Riedmiller, M. (2014) · HAL · Cited by 3,275
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Sutton, R. S., Precup, D., & Singh, S. (1999) · Artificial Intelligence · doi:10.1016/s0004-3702(99)00052-1 · Cited by 3,119
Konda, V. R., & Tsitsiklis, J. N. (2002) · Cited by 2,202
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161 · cited 12×
“In reinforcement learning, options (Sutton, Precup, and Singh 1999; Precup 2000) provide a framework for defin-ing such courses of action and for seamlessly learning and planning with them.”
BackgroundInfluentialOn-line Q-learning Using Connectionist Systems
Rummery, G. A. (1994) · Cited by 1,477
Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition
Dietterich, T. G. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.639 · Cited by 1,435
Dyna, an Integrated Architecture for Learning, Planning, and Reacting
Sutton, R. S. (1991) · ACM SIGART Bulletin · doi:10.1145/122344.122377 · Cited by 777 · cited 1×
“Classical dynamic programming methods such as value iteration and policy iteration [Puterman, 1994] or Dyna planning [Sutton, 1991; Sorg and Singh, 2010] can be used to provide a solution.”
MethodHierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06057 · Cited by 770
Temporal credit assignment in reinforcement learning
Sutton, R. S. (1984) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 749
Automatic Discovery of Subgoals in Reinforcement Learning using Diverse Density
McGovern, A., & Barto, A. G. (2001) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 418
Simulation and the Monte Carlo Method (Wiley Series in Probability and Statistics)
Rubinstein, R. Y., & Kroese, D. P. (1981) · Cited by 309
Peters, J., Vijayakumar, S., & Schaal, S. (2005) · Lecture Notes in Computer Science · doi:10.1007/11564096_29 · Cited by 307
Learning Options in Reinforcement Learning
Stolle, M., & Precup, D. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-45622-8_16 · Cited by 261
Self-citationTemporal abstraction in reinforcement learning
Precup, D., & Sutton, R. S. (2000) · Cited by 236
Self-citationSkill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining
Konidaris, G., & Barreto, A. S. (2009) · Cited by 227
Q-Cut—Dynamic Discovery of Sub-goals in Reinforcement Learning
Menache, I., Mannor, S., & Shimkin, N. (2002) · Lecture Notes in Computer Science · doi:10.1007/3-540-36755-1_25 · Cited by 218
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2012) · arXiv · doi:10.48550/arxiv.1207.4708 · Cited by 187
Using relative novelty to identify useful temporal abstractions in reinforcement learning
Şimşek, Ö., & Barto, A. G. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015353 · Cited by 176
Skill Characterization Based on Betweenness
Şimşek, Ö., & Barreto, A. S. (2008) · neural information processing systems · Cited by 127
Convergence rate of linear two-time-scale stochastic approximation
Konda, V. R., & Tsitsiklis, J. N. (2004) · The Annals of Applied Probability · doi:10.1214/105051604000000116 · Cited by 112
Probabilistic inference for determining options in reinforcement learning
Daniel, C., van Hoof, H., Peters, J., & Neumann, G. M. (2016) · Machine Learning · doi:10.1007/s10994-016-5580-x · Cited by 95
Bias in Natural Actor-Critic Algorithms
Thomas, P. S. (2014) · Cited by 93
Autonomous Skill Acquisition on a Mobile Manipulator
Konidaris, G., Kuindersma, S., Grupen, R. A., & Barto, A. G. (2011) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v25i1.7982 · Cited by 64
Adaptive Skills, Adaptive Partitions (ASAP)Preprint
Mankowitz, D. J., Mann, T., & Mannor, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.03351 · Cited by 53
Approximate Value Iteration with Temporally Extended Actions
Mann, T., Mannor, S., & Precup, D. (2015) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4676 · Cited by 50
Self-citationLinear Off-Policy Actor-Critic.
Degris, T., White, M., & Sutton, R. S. (2012) · International Conference on Machine Learning · Cited by 42
Compositional Planning Using Optimal Option ModelsPreprint
Silver, D., & Ciosek, K. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6473 · Cited by 40
Optimal policy switching algorithms for reinforcement learning
Comanici, G., & Precup, D. (2010) · Adaptive Agents and Multi-Agents Systems · doi:10.5555/1838206.1838300 · Cited by 30
Self-citationUnified Inter and Intra Options Learning Using Policy Gradient Methods
Levy, K. Y., & Shimkin, N. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-29946-9_17 · Cited by 26
Hierarchical Reinforcement Learning using Spatio-Temporal Abstractions and Deep Neural Networks.Preprint
Krishnamurthy, R., Lakshminarayanan, A. S., Kumar, P., & Ravindran, B. (2016) · arXiv (Cornell University) · Cited by 20
Sorg, J., & Singh, S. P. (2010) · Adaptive Agents and Multi-Agents Systems · doi:10.5555/1838206.1838211 · Cited by 20
Semantically Grounded Learning from Unstructured Demonstrations
Niekum, S. (2013) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · doi:10.7275/rhrr-5453 · Cited by 12
Cited by
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733
“Several works in DRL have attempted HRL by the use of one top-level policy that chooses between subpolicies, where the division of states or goals in to subpolicies is achieved either manually [1, 121, 54] or automatically [2, 129, 130].”
A Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
Deep Reinforcement Learning: A Survey
Wang, X., Wang, S., Liang, X., Zhao, D., Huang, J., Xu, X., Dai, B., & Miao, Q. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3207346 · Cited by 972
Lieder, F., & Griffiths, T. L. (2020) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1900061x · Cited by 892
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
Metadata
- Authors
- Bacon, P.-L., Harb, J., & Precup, D.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2017-02-13
- Volume
- 31
- Issue
- 1
- ISSN
- 2374-3468
- 2159-5399
- OpenAlex
- W2964227312
- Cited by
- 687
- References
- 39
- FWCI
- 33.46
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.