The DOI matches its Crossref record; 74 of the 75 references it cites were found in a registry or an index.
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Open accessOverview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In robotics, skills are often modelled using the Option framework [78].”
From Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects (2023) · cites this paper as [78]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,268
Axten, N., Newell, A., & Simon, H. A. (1973) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2063712 · Cited by 6,886
Newell, A. (1972) · Cited by 6,222
Dynamic Programming and Markov Processes.
Freimer, M. L., & Howard, R. A. (1961) · Journal of the American Statistical Association · doi:10.2307/2282268 · Cited by 3,388
A qualitative physics based on confluences
de Kleer, J., & Brown, J. S. (1984) · Artificial Intelligence · doi:10.1016/0004-3702(84)90037-7 · Cited by 1,510
Planning in a hierarchy of abstraction spaces
Sacerdoti, E. D. (1974) · Artificial Intelligence · doi:10.1016/0004-3702(74)90026-5 · Cited by 1,185
Learning to act using real-time dynamic programming
Barto, A. G., Bradtke, S. J., & Singh, S. P. (1995) · Artificial Intelligence · doi:10.1016/0004-3702(94)00011-o · Cited by 1,129
Learning and executing generalized robot plans
Fikes, R. E., Hart, P. E., & Nilsson, N. J. (1972) · Artificial Intelligence · doi:10.1016/0004-3702(72)90051-3 · Cited by 1,025
Reinforcement learning for robots using neural networks
Lin, L.-J. (1992) · Defense Technical Information Center (DTIC) · Cited by 841
On the Convergence of Stochastic Iterative Dynamic Programming Algorithms
Jaakkola, T., Jordan, M. I., & Singh, S. P. (1994) · Neural Computation · doi:10.1162/neco.1994.6.6.1185 · Cited by 801
Research works printed but not linked: 8
- E.G. Araujo, R.A. Grupen, Learning control composition in a complex environment, in: Proc. 4th International Conference on Simulation of Adaptive Behavior, 1996, pp. 333-342.
- S. Mahadevan, N. Marchalleck, T. Das, A. Gosavi, Self-improving factory simulation using continuous-time average-reward reinforcement learning, in: Proc. 14th International Conference on Machine Learning, 1997, pp. 202-210.
- D. Precup, R.S. Sutton, Multi-time models for reinforcement learning, in: Proc. ICML'97 Workshop on Modeling in Reinforcement Learning, 1997.
- D. Precup, R.S. Sutton, S.P. Singh, Planning with closed-loop macro actions, in: Working Notes 1997 AAAI Fall Symposium on Model-directed Autonomous Systems, 1997, pp. 70-76.
- S. Sastry, Algorithms for design of hybrid systems, in: Proc. International Conference of Information Sciences, 1997.
- S.P. Singh, Scaling reinforcement learning by learning variable temporal resolution models, in: Proc. 9th International Conference on Machine Learning, Morgan Kaufmann, San Mateo, CA, 1992, pp. 406-415.
- R.S. Sutton, B. Pinette, The learning of world models by connectionist networks, in: Proc. 7th Annual Conference of the Cognitive Science Society, 1985, pp. 54-64.
- C.J.C.H. Watkins, Learning with delayed rewards, Ph.D. Thesis, Cambridge University, 1989.
Cited by
Sorted by citations
Items 901–1,000 of 1,012
Thomas, P. S., & Barto, A. G. (2012) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2012.6400845 · Cited by 18
Multi-time-scale Markov decision processes for organizational decision-making
Wernz, C. (2013) · EURO Journal on Decision Processes · doi:10.1007/s40070-013-0020-7 · Cited by 18
A Continuous-Time Markov Decision Process-Based Method With Application in a Pursuit-Evasion Example
Jia, S., Wang, X., & Shen, L. (2015) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2015.2478875 · Cited by 18
Hierarchy Through Composition with Multitask LMDPs
Saxe, A., Earle, A. C., & Rosman, B. (2017) · International Conference on Machine Learning · Cited by 18
Sample and Feedback Efficient Hierarchical Reinforcement Learning from Human Preferences
Pinsler, R., Akrour, R., Osa, T., Peters, J., & Neumann, G. M. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460907 · Cited by 21
Takanobu, R., Zhuang, T., Huang, M., Feng, J., Tang, H., & Zheng, B. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308558.3313455 · Cited by 17
Dynamic Real-time Multimodal Routing with Hierarchical Hybrid Planning
Choudhury, S., Knickerbocker, J. P., & Kochenderfer, M. J. (2019) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2019.8814018 · Cited by 18
Planning with State Abstractions for Non-Markovian Task Specifications
Oh, Y., Patel, R., Nguyen, T. N., Huang, B., Pavlick, E., & Tellex, S. (2019) · Robotics: Science and Systems · doi:10.15607/rss.2019.xv.059 · Cited by 18
Jiang, N., Jin, S., & Zhang, C. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.06.024 · Cited by 18
Value Preserving State-Action Abstractions
Abel, D. L., Umbanhowar, N., Khetarpal, K., Arumugam, D., Precup, D., & Littman, M. L. (2020) · International Conference on Artificial Intelligence and Statistics · Cited by 18
Reddy, P. P., & Veloso, M. (2013) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v27i1.8481 · Cited by 18
McKee, K. R., Leibo, J. Z., Beattie, C., & Everett, R. (2022) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-022-09548-8 · Cited by 25
Behavioral Hierarchy: Exploration and Representation
Barto, A. G., Konidaris, G., & Vigorito, C. M. (2013) · Springer eBooks · doi:10.1007/978-3-642-39875-9_2 · Cited by 18
Logical Markov Decision Programs
Kristian Kersting, B., & Raedt, L. D. (2003) · Cited by 18
Learning and control of exploration primitives
Gordon, G., Fonio, E., & Ahissar, E. (2014) · Journal of Computational Neuroscience · doi:10.1007/s10827-014-0500-1 · Cited by 18
TD(0) converges provably faster than the residual gradient algorithm
Schoknecht, R., & Merke, A. (2003) · Cited by 18
From Single-Agent to Multi-Agent Reinforcement Learning: Foundational Concepts and Methods
Neto, G. (2005) · Cited by 18
Combo-Action: Training Agent For FPS Game with Auxiliary Tasks
Huang, S., Su, H., Zhu, J., & Chen, T. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301954 · Cited by 18
Porta, J. M., & Celaya, E. (2005) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1437 · Cited by 18
Deep Reinforcement Learning for Dexterous Manipulation with Concept NetworksPreprint
Gudimella, A., Story, R., Shaker, M., Kong, R., Brown, M., Shnayder, V., & Campos, M. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.06977 · Cited by 23
Data-efficient Hindsight Off-policy Option LearningPreprint
Wulfmeier, M., Rao, D., Hafner, R., Lampe, T., Abdolmaleki, A., Hertweck, T., Neunert, M., Tirumala, D., Siegel, N., Heess, N., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.15588 · Cited by 18
Behavior Priors for Efficient Reinforcement LearningPreprint
Tirumala, D., Galashov, A., Noh, H., Hasenclever, L., Pascanu, R., Schwarz, J., Desjardins, G., Czarnecki, W. M., Ahuja, A., Teh, Y. W., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14274 · Cited by 33
A Survey on Interactive Reinforcement Learning: Design Principles and Open ChallengesPreprint
Cruz, C. A., & Igarashi, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.12949 · Cited by 21
Behavioral building blocks for autonomous agents: description, identification, and learning
Barto, A. G., & Şimşek, Ö. (2008) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 18
Advances and applications in inverse reinforcement learning: a comprehensive review
Deshpande, S., Walambe, R. A., Kotecha, K. V., Selvachandran, G., & Abraham, A. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11100-0 · Cited by 17
Efficient Reinforcement Learning with Hierarchies of Machines by Leveraging Internal Transitions
Bai, A., & Russell, S. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/196 · Cited by 19
Recent advances in leveraging human guidance for sequential decision-making tasks
Zhang, R., Torabi, F., Warnell, G., & Stone, P. (2021) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-021-09514-w · Cited by 20
Relational Deep Reinforcement Learning for Routing in Wireless Networks
Manfredi, V., Wolfe, A. P., Wang, B., & Zhang, X. (2021) · IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM) · doi:10.1109/wowmom51794.2021.00029 · Cited by 17
Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulation
Liang, J., Sharma, M. K., LaGrassa, A., Vats, S., Saxena, S., & Kroemer, O. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9811575 · Cited by 17
Guo, Y., Zhang, Q., Wang, J., & Liu, S. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534349 · Cited by 17
Hierarchical goals contextualize local reward decomposition explanations
Rietz, F., Magg, S., Heintz, F., Stoyanov, T., Wermter, S., & Stork, J. A. (2022) · Neural Computing and Applications · doi:10.1007/s00521-022-07280-8 · Cited by 17
Exploring Potential Energy Surfaces Using Reinforcement Machine Learning
Mills, A. W., Goings, J. J., Beck, D. A. C., Yang, C., & Li, X. (2022) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.2c00373 · Cited by 18
Zhou, Y., & Ho, H. W. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105152 · Cited by 18
Human-Flow-Aware Long-Term Mobile Robot Task Planning Based on Hierarchical Reinforcement Learning
Liu, Y., Palmieri, L., Georgievski, I., & Aiello, M. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3280816 · Cited by 17
A Better Match for Drivers and Riders: Reinforcement Learning at Lyft
Azagirre, X., Balwally, A., Candeli, G., Chamandy, N., Han, B., King, A., Lee, H., Lončarić, M., Martin, S., Narasiman, V., et al. (2024) · INFORMS Journal on Applied Analytics · doi:10.1287/inte.2023.0083 · Cited by 17
Hierarchical control over foraging behavior by anterior cingulate cortex
Alejandro, R. J., & Holroyd, C. B. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2024.105623 · Cited by 18
Complex behavior from intrinsic motivation to occupy future action-state path space
Ramírez‐Ruiz, J., Grytskyy, D., Mastrogiuseppe, C., Habib, Y., & Moreno‐Bote, R. (2024) · Nature Communications · doi:10.1038/s41467-024-49711-1 · Cited by 31
Zhao, S., Wei, Y., Cheng, Y., & Li, Y. (2025) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2025.110855 · Cited by 17
Simple Local Models for Complex Dynamical Systems
Talvitie, E., & Singh, S. P. (2008) · Cited by 17
The two-dimensional organization of behavior
Ring, M. B., Schaul, T., & Schmidhuber, J. (2011) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2011.6037326 · Cited by 17
Efficient state-space modularization for planning: theory, behavioral and neural signatures
McNamee, D., Wolpert, D. M., & Lengyel, M. (2016) · Cambridge University Engineering Department Publications Database · Cited by 17
Offline reinforcement learning with task hierarchies
Schwab, D., & Ray, S. (2017) · Machine Learning · doi:10.1007/s10994-017-5650-8 · Cited by 18
Sperati, V., & Baldassarre, G. (2017) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2017.2772908 · Cited by 17
Arumugam, D., Karamcheti, S., Gopalan, N., Williams, E., Rhee, M., Wong, L. L. S., & Tellex, S. (2018) · Autonomous Robots · doi:10.1007/s10514-018-9792-8 · Cited by 20
Kim, W., Lee, C., & Kim, H. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594476 · Cited by 18
Learning With Options That Terminate Off-Policy
Harutyunyan, A., Vrancx, P., Bacon, P., Precup, D., & Nowé, A. (2018) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v32i1.11740 · Cited by 17
Hafez, M. B., Weber, C., Kerzel, M., & Wermter, S. (2020) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2020.103630 · Cited by 18
Automatic Skill Acquisition in Reinforcement Learning Agents Using Connection Bridge Centrality
Moradi, P., Shiri, M. E., & Entezari, N. (2010) · Communications in computer and information science · doi:10.1007/978-3-642-17604-3_6 · Cited by 17
Simulation-based Optimization of Resource Placement and Emergency Response
Bjarnason, R., Tadepalli, P. V., Fern, A., & Niedner, C. (2009) · Cited by 17
Automatic Hierarchical Reinforcement Learning for Efficient Large-Scale Service Composition
Wang, H., Huang, G., & Yu, Q. (2016) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws.2016.17 · Cited by 17
A Bee Algorithm for Multi-Agent Systems: Recruitment and Navigation Combined
Lemmens, N., Jong, S. D., Tuyls, K., & Nowé, A. (2007) · Cited by 17
Negative Update Intervals in Deep Multi-Agent Reinforcement Learning
Palmer, G., Savani, R., & Tuyls, K. (2019) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/afte1429 · Cited by 17
Robot Controllers for Highly Dynamic Environments with Real-time Constraints
Ferrein, A. (2010) · Künstliche Intell. · doi:10.1007/s13218-010-0041-3 · Cited by 17
Symmetry Learning for Function Approximation in Reinforcement LearningPreprint
Mahajan, A., & Tulabandhula, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02999 · Cited by 17
Idiosyncrasies and challenges of data driven learning in electronic tradingPreprint
Bacoyannis, V., Glukhov, V., Jin, T., Kochems, J., & Song, D. R. (2018) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1811.09549 · Cited by 17
Importance Resampling for Off-policy PredictionPreprint
Schlegel, M., Chung, W., Graves, D., Qian, J., & White, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04328 · Cited by 17
A Boolean Task Algebra for Reinforcement LearningPreprint
Tasse, G. N., James, S. D., & Rosman, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.01394 · Cited by 17
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations
Mandlekar, A., Xu, D., Martín-Martín, R., Savarese, S., & Fei-Fei, L. (2020) · arXiv · doi:10.48550/arxiv.2003.06085 · Cited by 17
The Naive Utility Calculus as a unified, quantitative framework for action understandingPreprint
Jara‐Ettinger, J., Schulz, L., & Tenenbaum, J. B. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/e8xsv · Cited by 21
Li, M., Cai, K., & Zhao, P. (2024) · Transportation Research Part E Logistics and Transportation Review · doi:10.1016/j.tre.2024.103878 · Cited by 16
Asynchronous multi-agent deep reinforcement learning under partial observability
Xiao, Y., Tan, W., Hoffman, J., Xia, T., & Amato, C. (2025) · The International Journal of Robotics Research · doi:10.1177/02783649241306124 · Cited by 18
Dexterous Manipulation Through Imitation Learning: A Survey
An, S., Meng, Z., Tang, C., Zhou, Y., Liu, T., Ding, F., Zhang, S., Mu, Y., Song, R., Zhang, W., et al. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3646183 · Cited by 56
A Novelty-Centric Agent Architecture for Changing Worlds.
Faizan, M., Sarathy, V., Tatiya, G., Goel, S., Gyawali, S., Castro, M. G., Sinapov, J., & Scheutz, M. J. (2021) · Autonomous Agents and Multi-Agent Systems · Cited by 16
Adaptive heterogeneous multi-robot collaboration from formal task specifications
Schillinger, P., García, S., Makris, A., Roditakis, K., Logothetis, M., Alevizos, K., Ren, W., Tajvar, P., Pelliccione, P., Argyros, A., et al. (2021) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2021.103866 · Cited by 16
From Adaptive Locomotion to Predictive Action Selection – Cognitive Control for a Six-Legged Walker
Schilling, M., Paskarbeit, J., Ritter, H. J., Schneider, A., & Cruse, H. (2021) · IEEE Transactions on Robotics · doi:10.1109/tro.2021.3106832 · Cited by 15
Opportunity cost determines free-operant action initiation latency and predicts apathy
Nair, A., Niyogi, R. K., Shang, F., Tabrizi, S., Rees, G., & Rutledge, R. (2021) · Psychological Medicine · doi:10.1017/s0033291721003469 · Cited by 12
Xu, M., Yue, P., Yu, F., Yang, C., Zhang, M., Li, S., & Li, H. (2022) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2022.2119477 · Cited by 16
Mice identify subgoal locations through an action-driven mapping process
Shamash, P., Lee, S., Saxe, A., & Branco, T. (2023) · Neuron · doi:10.1016/j.neuron.2023.03.034 · Cited by 14
Huang, Y., & Chin, K. (2023) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2023.3282984 · Cited by 16
State Super Sampling Soft Actor–Critic Algorithm for Multi-AUV Hunting in 3D Underwater Environment
Wang, Z., Sui, Y., Qin, H., & Lu, H. (2023) · Journal of Marine Science and Engineering · doi:10.3390/jmse11071257 · Cited by 16
STAP: Sequencing Task-Agnostic Policies
Agia, C., Migimatsu, T., Wu, J., & Bohg, J. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160220 · Cited by 16
Reward-respecting subtasks for model-based reinforcement learning
Sutton, R. S., Machado, M. C., Holland, G. Z., Szepesvári, D., Timbers, F., Tanner, B. K., & White, A. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.104001 · Cited by 25
Decentralized multi-agent reinforcement learning based on best-response policies
Gabler, V., & Wollherr, D. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1229026 · Cited by 17
Zhang, D., Meng, H., Wang, M., Xu, X., Yan, J., & Li, X. (2024) · Transactions in GIS · doi:10.1111/tgis.13252 · Cited by 16
Machine learning for interactive systems and robots
Cuayáhuitl, H., van Otterlo, M., Dethlefs, N., & Frommberger, L. (2013) · doi:10.1145/2493525.2493530 · Cited by 17
Continually Learning Optimal Allocations of Services to Tasks
Achbany, Y., Jureta, I., Faulkner, S., & Fouss, F. (2008) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2008.12 · Cited by 16
Structural knowledge transfer by spatial abstraction for reinforcement learning agents
Frommberger, L., & Wolter, D. (2010) · Adaptive Behavior · doi:10.1177/1059712310391484 · Cited by 16
Prescriptive Analytics Using Synthetic Information
MARATHE, M. V., Mortveit, H. S., Parikh, N., & Swarup, S. (2014) · Advances in data mining and database management book series · doi:10.4018/978-1-4666-5063-3.ch001 · Cited by 16
Assurance in Reinforcement Learning Using Quantitative Verification
Mason, G., Călinescu, R., Kudenko⋆, D., & Banks, A. (2017) · Smart innovation, systems and technologies · doi:10.1007/978-3-319-66790-4_5 · Cited by 15
Computational approaches to habits in a model-free world
Pauli, W. M., Cockburn, J., Pool, E. R., Perez, O. D., & O’Doherty, J. P. (2017) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2017.12.001 · Cited by 18
On Value Function Representation of Long Horizon Problems
Lehnert, L., Laroche, R., & van Seijen, H. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11646 · Cited by 16
Autonomous Identification and Goal-Directed Invocation of Event-Predictive Behavioral Primitives
Gumbsch, C., Butz, M. V., & Martius, G. (2019) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2019.2925890 · Cited by 15
ACE: An Actor Ensemble Algorithm for Continuous Control with Tree Search
Zhang, S., & Yao, H. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015789 · Cited by 20
Barekatain, M., Yonetani, R., & Hamaya, M. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/430 · Cited by 17
Li, S., Wei, C., Yan, X., Ma, L., Chen, D., & Wang, Y. (2020) · IEEE Access · doi:10.1109/access.2020.2974885 · Cited by 18
Resource-Driven Mission-Phasing Techniques for Constrained Agents in Stochastic Environments
Wu, J., & Durfee, E. H. (2013) · Cited by 16
Winning the L2RPN Challenge: Power Grid Management via Semi-Markov Afterstate Actor-Critic
Yoon, D., Hong, S., Lee, B.-J., & Kim, K.-E. (2021) · Cited by 16
Metrics and Continuity in Reinforcement Learning
Lan, C. L., Bellemare, M. G., & Castro, P. S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i9.17005 · Cited by 16
A peer-to-peer market for utility exchanges in Eco-Industrial Parks using automated negotiations
Kröhling, D. E., Mione, F. M., Hernández, F., & Martínez, E. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.116211 · Cited by 20
Liu, R. G., & Frank, M. J. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103770 · Cited by 15
Decision tree methods for finding reusable MDP homomorphisms
Wolfe, A. P., & Barto, A. G. (2006) · Cited by 16
Multiagent meta-level control for radar coordination
Cheng, S., Raja, A., & Lesser, V. (2013) · Web Intelligence and Agent Systems An International Journal · doi:10.3233/wia-130264 · Cited by 16
Predictive Representations: Building Blocks of Intelligence
Carvalho, W., Tomov, M. S., de Cothi, W., Barry, C., & Gershman, S. J. (2024) · Neural Computation · doi:10.1162/neco_a_01705 · Cited by 15
Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement Learning Approach
Hu, B., Zhao, C., Zhang, P., Zhou, Z., Yang, Y., Xu, Z., & Liu, B. (2023) · arXiv · doi:10.48550/arxiv.2306.03604 · Cited by 15
“In the remainder of this paper, we begin by formalizing the problem based on the options framework [15], [16], and discussing related works in Section II.”
BackgroundMethodHierarchical control using networks trained with higher-level forward models
Wayne, G., & Abbott, L. F. (2014) · Neural Computation · doi:10.1162/neco_a_00639 · Cited by 14
Learning Interactive Real-World Simulators
Yang, S., Du, Y., Ghasemipour, K., Tompson, J., Kaelbling, L., Schuurmans, D., & Abbeel, P. (2023) · arXiv · doi:10.48550/arxiv.2310.06114 · Cited by 346
“The history conditioned base model operates at temporal and spatial resolution [16 , 24 , 40] , and the two spatial super-resolution models operate at spatial resolution [24 , 40] → [48 , 80] and [48 , 80] → [192 , 320] , respectively.”
BackgroundAction chunking as conditional policy compression
Lai, L., Huang, A. Z., & Gershman, S. J. (2022) · doi:10.31234/osf.io/z8yrv · Cited by 9
CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments
Liu, X., Paul, S., Chatterjee, M., & Cherian, A. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i4.28167 · Cited by 6
“Putting it all together, our objective to learn these policies Here, b ′ is the updated belief and Z ′ is the multi-time transition function (Sutton, Precup, and Singh 1999) given by: Below, we detail the architecture of each of these policies.”
BackgroundSkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
Li, X., Liu, Y., Chen, W., You, B., Di, Z., He, Y., Zheng, S., Choe, K. W., Sun, J., Wang, S., et al. (2026) · arXiv · doi:10.48550/arxiv.2602.12670 · Cited by 1
“This modular approach builds on the options framework for temporal abstraction (Sutton et al., 1999) and cognitive architectures for language agents (Sumers et al., 2023), mirroring successful computing paradigms: foundation models provide base capabilities (analogous to CPUs), agent harnesses…”
Leveraging Sequentiality in Reinforcement Learning from a Single Demonstration
Chenu, A., Serris, O., Sigaud, O., & Perrin-Gilbert, N. (2022) · arXiv · doi:10.48550/arxiv.2211.04786
“In Option-based HRL [38, 39], an option contains: 1) a set of initial states where the option can be activated, 2) a termination function deciding if the option should be terminated given the current state, 3) the intra-option policy which controls the agent at a single time step scale and, 4) the policy over options…”
Influential
Metadata
- Authors
- Sutton, R. S., Precup, D., & Singh, S.
- Affiliations
- AT&T (United States) (US)
- University of Massachusetts Amherst (US)
- Corresponding author
- Sutton, Richard S.
- Venue
- Artificial Intelligence
- Publisher
- Elsevier BV
- Published
- 1999-08
- Volume
- 112
- Issue
- 1-2
- Pages
- 181-211
- ISSN
- 0004-3702 (ISSN-L)
- OpenAlex
- W2109910161
- Cited by
- 3,119
- References
- 75
- Citations by year
- 2026: 179 · 2025: 206 · 2024: 188 · 2023: 212 · 2022: 213 · 2021: 270 · 2020: 303 · 2019: 238 · 2018: 183 · 2017: 112 · 2016: 99 · 2015: 80 · 2014: 74 · 2013: 91 · 2012: 86
- FWCI
- 27.32
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Artificial Intelligence (publishedVersion)
- Funders
- National Science Foundation (9711753, 9511805)
- Air Force Office of Scientific Research (F49620)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- temporal abstraction, options framework, semi-Markov decision process, Markov decision process, hierarchical reinforcement learning, Q-learning, dynamic programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
