The DOI matches its Crossref record; 74 of the 75 references it cites were found in a registry or an index.
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Open accessOverview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In robotics, skills are often modelled using the Option framework [78].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [78]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,268
Axten, N., Newell, A., & Simon, H. A. (1973) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2063712 · Cited by 6,886
Newell, A. (1972) · Cited by 6,222
Dynamic Programming and Markov Processes.
Freimer, M. L., & Howard, R. A. (1961) · Journal of the American Statistical Association · doi:10.2307/2282268 · Cited by 3,388
A qualitative physics based on confluences
de Kleer, J., & Brown, J. S. (1984) · Artificial Intelligence · doi:10.1016/0004-3702(84)90037-7 · Cited by 1,510
Planning in a hierarchy of abstraction spaces
Sacerdoti, E. D. (1974) · Artificial Intelligence · doi:10.1016/0004-3702(74)90026-5 · Cited by 1,185
Learning to act using real-time dynamic programming
Barto, A. G., Bradtke, S. J., & Singh, S. P. (1995) · Artificial Intelligence · doi:10.1016/0004-3702(94)00011-o · Cited by 1,129
Learning and executing generalized robot plans
Fikes, R. E., Hart, P. E., & Nilsson, N. J. (1972) · Artificial Intelligence · doi:10.1016/0004-3702(72)90051-3 · Cited by 1,025
Reinforcement learning for robots using neural networks
Lin, L.-J. (1992) · Defense Technical Information Center (DTIC) · Cited by 841
On the Convergence of Stochastic Iterative Dynamic Programming Algorithms
Jaakkola, T., Jordan, M. I., & Singh, S. P. (1994) · Neural Computation · doi:10.1162/neco.1994.6.6.1185 · Cited by 801
Temporal Difference Learning and TD-Gammon
Tesauro, G. (1995) · ICGA Journal · doi:10.3233/icg-1995-18207 · Cited by 784
Improving generalization for temporal difference learning: The successor representation
Dayan, P. (1993) · Neural Computation · doi:10.1162/neco.1993.5.4.613 · Cited by 693
Reinforcement Learning with Hierarchies of Machines
Parr, R., & Russell, S. (1997) · Cited by 618
Automatic programming of behavior-based robots using reinforcement learning
Mahadevan, S., & Connell, J. H. (1992) · Artificial Intelligence · doi:10.1016/0004-3702(92)90058-6 · Cited by 583
Dayan, P., & Hinton, G. E. (1992) · Cited by 540
Grossman, R. L., Nerode, A., Ravn, A. P., & Rischel, H. (1993) · Cited by 512
Chunking in Soar: The Anatomy of a General Learning Mechanism
Laird, J. E., Rosenbloom, P. S., & Newell, A. (1986) · Machine Learning · doi:10.1023/a:1022639103969 · Cited by 509
Improving Elevator Performance Using Reinforcement Learning
Crites, R. H., & Barto, A. G. (1995) · Cited by 499
Probabilistic robot navigation in partially observable environments
Simmons, R. G., & Koenig, S. (1995) · Cited by 482
Behaviour-based control: examples from navigation, learning, and group behaviour
Matarić, M. J. (1997) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/095281397147149 · Cited by 434
Hybrid Models for Motion Control Systems
Brockett, R. W. (1993) · Birkhäuser Boston eBooks · doi:10.1007/978-1-4612-0313-1_2 · Cited by 378
Learning to coordinate behaviors
Maes, P., & Brooks, R. A. (1990) · Cited by 376
Quantitative results concerning the utility of explanation-based learning
Minton, S. (1990) · Artificial Intelligence · doi:10.1016/0004-3702(90)90059-9 · Cited by 372
Teleo-Reactive Programs for Agent Control
Nilsson, N. J. (1994) · Journal of Artificial Intelligence Research · doi:10.1613/jair.30 · Cited by 367
Made-up minds: a constructivist approach to artificial intelligence
Drescher, G. L. (1992) · Choice Reviews Online · doi:10.5860/choice.29-3935 · Cited by 315
Transfer of Learning by Composing Solutions of Elemental Sequential Tasks
Singh, S. P. (1992) · Machine Learning · doi:10.1007/bf00992700 · Cited by 301
The MAXQ Method for Hierarchical Reinforcement Learning
Dietterich, T. G. (1998) · Cited by 285
Purposive behavior acquisition for a real robot by vision-based reinforcement learning
Asada, M., Noda, S., Tawaratsumida, S., & Hosoda, K. (1996) · Machine Learning · doi:10.1007/bf00117447 · Cited by 274
Reinforcement Learning Methods for Continuous-Time Markov Decision Problems
Bradtke, S. J., & Duff, M. O. (1994) · Cited by 263
Moore, A., & Atkeson, C. G. (1995) · Machine Learning · doi:10.1023/a:1022656217772 · Cited by 261
Robot shaping: developing autonomous agents through learning
Dorigo, M., & Colombetti, M. (1994) · Artificial Intelligence · doi:10.1016/0004-3702(94)90047-7 · Cited by 238
Hierarchical Solution of Markov Decision Processes using Macro-actionsPreprint
Hauskrecht, M., Meuleau, N., Kaelbling, L. P., Dean, T., & Boutilier, C. E. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.7381 · Cited by 230
Hierarchical control and learning for markov decision processes
Parr, R., & Russell, S. (1998) · Cited by 221
Reinforcement Learning for Dynamic Channel Allocation in Cellular Telephone Systems
Singh, S. P., & Bertsekas, D. P. (1996) · Cited by 214
Planning under time constraints in stochastic domains
Dean, T., Kaelbling, L. P., Kirman, J., & Nicholson, A. E. (1995) · Artificial Intelligence · doi:10.1016/0004-3702(94)00086-g · Cited by 201
A Q-learning-based dynamic channel assignment technique for mobile communication systems
Nie, J., & Haykin, S. (1999) · IEEE Transactions on Vehicular Technology · doi:10.1109/25.790549 · Cited by 177
Finding Structure in Reinforcement Learning
Thrun, S., & Schwartz, A. (1994) · Cited by 176
Solving very large weakly coupled Markov decision processes
Meuleau, N., Hauskrecht, M., Kim, K.-E., Peshkin, L., Kaelbling, L. P., Dean, T., & Boutilier, C. E. (1998) · Cited by 174
Learning Search Control Knowledge: An Explanation-Based Approach
Minton, S. (2011) · CERN Document Server (European Organization for Nuclear Research) · Cited by 172
Hierarchical Learning in Stochastic Domains: Preliminary Results
Kaelbling, L. P. (1993) · Elsevier eBooks · doi:10.1016/b978-1-55860-307-3.50028-9 · Cited by 171
Wiering, M., & Schmidhuber, J. (1997) · Adaptive Behavior · doi:10.1177/105971239700600202 · Cited by 167
TD Models: Modeling the World at a Mixture of Time Scales
Sutton, R. S. (1995) · Elsevier eBooks · doi:10.1016/b978-1-55860-377-6.50072-4 · Cited by 145
A Heuristic Approach to the Discovery of Macro-Operators
Iba, G. A. (1989) · Machine Learning · doi:10.1023/a:1022693717366 · Cited by 143
Decomposition Techniques for Planning in Stochastic Domains
Dean, T., & Lin, S. (1995) · Cited by 132
Intra-Option Learning about Temporally Abstract Actions
Sutton, R. S., Precup, D., & Singh, S. P. (1998) · Cited by 129
Learning to solve problems by searching for macro-operators
Korf, R. E. (1983) · Pitman Advanced Pub. Program eBooks · Cited by 119
Reinforcement learning with a hierarchy of abstract models
Singh, S. P. (1992) · Cited by 108
Multi-time Models for Temporally Abstract Planning
Precup, D., & Sutton, R. S. (1997) · Cited by 108
Theoretical results on reinforcement learning with temporally abstract options
Precup, D., Sutton, R. S., & Singh, S. P. (1998) · Lecture notes in computer science · doi:10.1007/bfb0026709 · Cited by 103
Behavior analysis and training-a methodology for behavior engineering
Colombetti, M., Dorigo, M., & Borghi, G. (1996) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/3477.499789 · Cited by 98
Behavior coordination for a mobile robot using modular reinforcement learning
Uchibe, E., Asada, M., & Hosoda, K. (2002) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.1996.568989 · Cited by 93
Automatic programming of robots using genetic programming
Koza, J. R., & Rice, J. P. (1992) · Cited by 90
Modeling agents as qualitative decision makers
Brafman, R. I., & Tennenholtz, M. (1997) · Artificial Intelligence · doi:10.1016/s0004-3702(97)00024-6 · Cited by 85
The Problem of Expensive Chunks and its Solution by Restricting Expressiveness
Tambe, M., Newell, A., & Rosenbloom, P. S. (1990) · Machine Learning · doi:10.1023/a:1022603523533 · Cited by 76
A feedback control structure for on-line learning tasks
Huber, M., & Grupen, R. A. (1997) · Robotics and Autonomous Systems · doi:10.1016/s0921-8890(97)00044-4 · Cited by 64
Boutilier, C. E., Brafman, R. I., & Geib, C. (1997) · Cited by 52
Etzioni, O. (1990) · National Conference on Artificial Intelligence · Cited by 51
Module-Based Reinforcement Learning: Experiments with a Real Robot
Kalmár, Z., Szepesvári, C., & Lörincz, A. (1998) · Machine Learning · doi:10.1023/a:1007440607681 · Cited by 49
Qualitative system identification: deriving structure from behavior
Say, A. C. C., & Kuru, S. (1996) · Artificial Intelligence · doi:10.1016/0004-3702(95)00016-x · Cited by 48
Robust Reinforcement Learning in Motion Planning
Singh, S. P., Barto, A. G., Grupen, R. A., & Connolly, C. I. (1993) · Cited by 48
Learning to Plan in Continuous Domains
DeJong, G. (1994) · Artificial Intelligence · doi:10.1016/0004-3702(94)90038-8 · Cited by 46
High-Level Planning and Control with Incomplete Information Using POMDP's
Bonet, B. (1998) · Cited by 45
A statistical approach to solving the EBL utility problem
Greiner, R., & Jurišica, I. (1992) · Cited by 44
Macro-Actions in Reinforcement Learning: An Empirical Analysis
McGovern, A., & Sutton, R. S. (1998) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 42
Commonsense knowledge of space: learning from experience
Kuipers, B. J. (1979) · Cited by 41
Yin, G. G., & Zhang, Q. (2012) · Stochastic modelling and applied probability · doi:10.1007/978-1-4614-4346-9_7 · Cited by 38
Improved Switching among Temporally Abstract Actions
Sutton, R. S., Singh, S. P., Precup, D., & Ravindran, B. (1998) · Cited by 33
The Efficient Learning of Multiple Task Sequences
Singh, S. P. (1991) · Cited by 32
Reinforcement Learning for Call Admission Control and Routing in Integrated Service Networks
Marbach, P., Mihatsch, O., Schulte, M., & Tsitsiklis, J. N. (1997) · Cited by 25
Ring, M. B. (1991) · Elsevier eBooks · doi:10.1016/b978-1-55860-200-7.50071-4 · Cited by 25
Composing functions to speed up reinforcement learning in a changing world
Drummond, C. (1998) · Lecture notes in computer science · doi:10.1007/bfb0026708 · Cited by 21
Scaling Reinforcement Learning Techniques via Modularity
Wixson, L. E. (1991) · Elsevier eBooks · doi:10.1016/b978-1-55860-200-7.50076-3 · Cited by 17
A statistical approach to adaptive problem solving
Gratch, J. S., & DEJONG, G. F. (1996) · Artificial Intelligence · doi:10.1016/s0004-3702(96)00011-2 · Cited by 11
Reasoning About Probabilistic Actions At Multiple Levels of Granularity
Chrisman, L. (1994) · Cited by 6
Research works printed but not linked: 8
- E.G. Araujo, R.A. Grupen, Learning control composition in a complex environment, in: Proc. 4th International Conference on Simulation of Adaptive Behavior, 1996, pp. 333-342.
- S. Mahadevan, N. Marchalleck, T. Das, A. Gosavi, Self-improving factory simulation using continuous-time average-reward reinforcement learning, in: Proc. 14th International Conference on Machine Learning, 1997, pp. 202-210.
- D. Precup, R.S. Sutton, Multi-time models for reinforcement learning, in: Proc. ICML'97 Workshop on Modeling in Reinforcement Learning, 1997.
- D. Precup, R.S. Sutton, S.P. Singh, Planning with closed-loop macro actions, in: Working Notes 1997 AAAI Fall Symposium on Model-directed Autonomous Systems, 1997, pp. 70-76.
- S. Sastry, Algorithms for design of hybrid systems, in: Proc. International Conference of Information Sciences, 1997.
- S.P. Singh, Scaling reinforcement learning by learning variable temporal resolution models, in: Proc. 9th International Conference on Machine Learning, Morgan Kaufmann, San Mateo, CA, 1992, pp. 406-415.
- R.S. Sutton, B. Pinette, The learning of world models by connectionist networks, in: Proc. 7th Annual Conference of the Cognitive Science Society, 1985, pp. 54-64.
- C.J.C.H. Watkins, Learning with delayed rewards, Ph.D. Thesis, Cambridge University, 1989.
Cited by
Sorted by citations
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733
Dynamic bayesian networks: representation, inference and learning
Murphy, K. P., & Russell, S. (2002) · Cited by 2,579
Deep reinforcement learning for autonomous driving: A survey
Kiran, B. R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A. A. A., Yogamani, S., & Perez, P. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3054625 · Cited by 2,342
Brochu, E., Cora, V. M., & de Freitas, N. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1012.2599 · Cited by 2,568
Metadata
- Authors
- Sutton, R. S., Precup, D., & Singh, S.
- Affiliations
- AT&T (United States) (US)
- University of Massachusetts Amherst (US)
- Corresponding author
- Sutton, Richard S.
- Venue
- Artificial Intelligence
- Publisher
- Elsevier BV
- Published
- 1999-08
- Volume
- 112
- Issue
- 1-2
- Pages
- 181-211
- ISSN
- 0004-3702 (ISSN-L)
- OpenAlex
- W2109910161
- Cited by
- 3,119
- References
- 75
- Citations by year
- 2026: 179 · 2025: 206 · 2024: 188 · 2023: 212 · 2022: 213 · 2021: 270 · 2020: 303 · 2019: 238 · 2018: 183 · 2017: 112 · 2016: 99 · 2015: 80 · 2014: 74 · 2013: 91 · 2012: 86
- FWCI
- 27.32
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Artificial Intelligence (publishedVersion)
- Funders
- National Science Foundation (9711753, 9511805)
- Air Force Office of Scientific Research (F49620)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- temporal abstraction, options framework, semi-Markov decision process, Markov decision process, hierarchical reinforcement learning, Q-learning, dynamic programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
