The DOI matches its Crossref record; 74 of the 75 references it cites were found in a registry or an index.
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Open accessOverview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In robotics, skills are often modelled using the Option framework [78].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [78]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,268
Axten, N., Newell, A., & Simon, H. A. (1973) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2063712 · Cited by 6,886
Newell, A. (1972) · Cited by 6,222
Dynamic Programming and Markov Processes.
Freimer, M. L., & Howard, R. A. (1961) · Journal of the American Statistical Association · doi:10.2307/2282268 · Cited by 3,388
A qualitative physics based on confluences
de Kleer, J., & Brown, J. S. (1984) · Artificial Intelligence · doi:10.1016/0004-3702(84)90037-7 · Cited by 1,510
Planning in a hierarchy of abstraction spaces
Sacerdoti, E. D. (1974) · Artificial Intelligence · doi:10.1016/0004-3702(74)90026-5 · Cited by 1,185
Learning to act using real-time dynamic programming
Barto, A. G., Bradtke, S. J., & Singh, S. P. (1995) · Artificial Intelligence · doi:10.1016/0004-3702(94)00011-o · Cited by 1,129
Learning and executing generalized robot plans
Fikes, R. E., Hart, P. E., & Nilsson, N. J. (1972) · Artificial Intelligence · doi:10.1016/0004-3702(72)90051-3 · Cited by 1,025
Reinforcement learning for robots using neural networks
Lin, L.-J. (1992) · Defense Technical Information Center (DTIC) · Cited by 841
On the Convergence of Stochastic Iterative Dynamic Programming Algorithms
Jaakkola, T., Jordan, M. I., & Singh, S. P. (1994) · Neural Computation · doi:10.1162/neco.1994.6.6.1185 · Cited by 801
Research works printed but not linked: 8
- E.G. Araujo, R.A. Grupen, Learning control composition in a complex environment, in: Proc. 4th International Conference on Simulation of Adaptive Behavior, 1996, pp. 333-342.
- S. Mahadevan, N. Marchalleck, T. Das, A. Gosavi, Self-improving factory simulation using continuous-time average-reward reinforcement learning, in: Proc. 14th International Conference on Machine Learning, 1997, pp. 202-210.
- D. Precup, R.S. Sutton, Multi-time models for reinforcement learning, in: Proc. ICML'97 Workshop on Modeling in Reinforcement Learning, 1997.
- D. Precup, R.S. Sutton, S.P. Singh, Planning with closed-loop macro actions, in: Working Notes 1997 AAAI Fall Symposium on Model-directed Autonomous Systems, 1997, pp. 70-76.
- S. Sastry, Algorithms for design of hybrid systems, in: Proc. International Conference of Information Sciences, 1997.
- S.P. Singh, Scaling reinforcement learning by learning variable temporal resolution models, in: Proc. 9th International Conference on Machine Learning, Morgan Kaufmann, San Mateo, CA, 1992, pp. 406-415.
- R.S. Sutton, B. Pinette, The learning of world models by connectionist networks, in: Proc. 7th Annual Conference of the Cognitive Science Society, 1985, pp. 54-64.
- C.J.C.H. Watkins, Learning with delayed rewards, Ph.D. Thesis, Cambridge University, 1989.
Cited by
Sorted by citations
Items 1,001–1,012 of 1,012
Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning
Luo, Z., Tessler, C., Lin, T., Yuan, Y., He, T., Xiao, W., Guo, Y., Chechik, G., Kitani, K., Fan, L., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.12278
Training a Team of Language Models as Options to Build an SQL-Based Memory
Lee, S., & Ko, H. (2025) · Applied Sciences · doi:10.3390/app152111399
Lyfe Agents: Generative agents for low-cost real-time social interactions
Kaiya, Z., Naim, M., Kondic, J., Cortes, M., Ge, J., Luo, S., Yang, G. R., & Ahn, A. (2023) · arXiv · doi:10.48550/arxiv.2310.02172 · Cited by 2
Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving
Shalev-Shwartz, S., Shammah, S., & Shashua, A. (2016) · arXiv · doi:10.48550/arxiv.1610.03295 · Cited by 509
Modular Multitask Reinforcement Learning with Policy Sketches
Andreas, J., Klein, D., & Levine, S. (2016) · arXiv · doi:10.48550/arxiv.1611.01796 · Cited by 274
A free energy principle for a particular physics
Friston, K. (2019) · arXiv · doi:10.48550/arxiv.1906.10184 · Cited by 222
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
Gupta, A., Kumar, V., Lynch, C., Levine, S., & Hausman, K. (2019) · arXiv · doi:10.48550/arxiv.1910.11956 · Cited by 153
Deep Reinforcement Learning at the Edge of the Statistical Precipice
Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., & Bellemare, M. G. (2021) · arXiv · doi:10.48550/arxiv.2108.13264 · Cited by 142
An empirical investigation of the challenges of real-world reinforcement learning
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2020) · arXiv · doi:10.48550/arxiv.2003.11881 · Cited by 78
Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration
Liu, E. Z., Guu, K., Pasupat, P., Shi, T., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1802.08802 · Cited by 47
Accelerating robotic reinforcement learning via parameterized action primitives
Dalal, M., Pathak, D., & Salakhutdinov, R. (2021) · arXiv · doi:10.48550/arxiv.2110.15360 · Cited by 44
AI Research Considerations for Human Existential Safety (ARCHES)
Critch, A., & Krueger, D. (2020) · arXiv · doi:10.48550/arxiv.2006.04948 · Cited by 25
Metadata
- Authors
- Sutton, R. S., Precup, D., & Singh, S.
- Affiliations
- AT&T (United States) (US)
- University of Massachusetts Amherst (US)
- Corresponding author
- Sutton, Richard S.
- Venue
- Artificial Intelligence
- Publisher
- Elsevier BV
- Published
- 1999-08
- Volume
- 112
- Issue
- 1-2
- Pages
- 181-211
- ISSN
- 0004-3702 (ISSN-L)
- OpenAlex
- W2109910161
- Cited by
- 3,119
- References
- 75
- Citations by year
- 2026: 179 · 2025: 206 · 2024: 188 · 2023: 212 · 2022: 213 · 2021: 270 · 2020: 303 · 2019: 238 · 2018: 183 · 2017: 112 · 2016: 99 · 2015: 80 · 2014: 74 · 2013: 91 · 2012: 86
- FWCI
- 27.32
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Artificial Intelligence (publishedVersion)
- Funders
- National Science Foundation (9711753, 9511805)
- Air Force Office of Scientific Research (F49620)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- temporal abstraction, options framework, semi-Markov decision process, Markov decision process, hierarchical reinforcement learning, Q-learning, dynamic programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
