CoChat
Sign Up
PreprintCited by 6Reinforcement Learning in RoboticsAI-based Problem Solving and PlanningArtificial Intelligence in Games

Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversity

Open access
Guan, Lin, Sreedharan, Sarath, Kambhampati, Subbarao·2022·arXiv
DOI 10.48550/arxiv.2202.02886arXiv 2202.02886
Open this paper in CoChat — free accountOpen PDF

Abstract

Creating reinforcement learning (RL) agents that are capable of accepting and leveraging task-specific knowledge from humans has been long identified as a possible strategy for developing scalable approaches for solving long-horizon problems. While previous works have looked at the possibility of using symbolic models along with RL approaches, they tend to assume that the high-level action models are executable at low level and the fluents can exclusively characterize all desirable MDP states. Symbolic models of real world tasks are however often incomplete. To this end, we introduce Approximate Symbolic-Model Guided Reinforcement Learning, wherein we will formalize the relationship between the symbolic model and the underlying MDP that will allow us to characterize the incompleteness of the symbolic model. We will use these models to extract high-level landmarks that will be used to decompose the task. At the low level, we learn a set of diverse policies for each possible task subgoal identified by the landmark, which are then stitched together. We evaluate our system by testing on three different benchmark domains and show how even with incomplete symbolic model information, our approach is able to discover the task structure and efficiently guide the RL agent towards the goal.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2202.02886

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Approximate symbolic models can help drastically improve performance.

    “the well known fact that even approximate symbolic models can help drastically improve the performance (c.f. ( Guan et al., 2022))”

    From LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks (2024) · cites this paper as Guan et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 40

  1. Achiam, J., Edwards, H., Amodei, D., and Abbeel, P. Variational option discovery algorithms. ArXiv , abs/1807.10299, 2018. · arXiv
  2. Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Ho, D., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jang, E., Ruano, R. J., Jeffrey, K., Jesmonth, S., Joshi, N., Julian, R., Kalashnikov, D., Kuang, Y ., Lee, K.-H., Levine, S., Lu, Y., Luu, L., Parada, C., Pastor, P., Quiambao, J., Rao, K., Rettinghouse, J., Reyes, D., Sermanet, P., Sievers, N., Tan, C., Toshev, A., Vanhoucke, V., Xia, F., Xiao, T., Xu, P., Xu, S., and Yan, M. Do as i can and not as i say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.01691 , 2022. · arXiv
  3. Andreas, J., Klein, D., and Levine, S. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning , pp. 166-175. PMLR, 2017.
  4. Basu, C., Singhal, M., and Dragan, A. D. Learning from richer human guidance: Augmenting comparison-based learning with feature queries. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction , pp. 132-140, 2018.
  5. Chakraborti, T., Sreedharan, S., Grover, S., and Kambhampati, S. Plan explanations as model reconciliation-an empirical study. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI) , pp. 258266. IEEE, 2019.
  6. De Giacomo, G., Iocchi, L., Favorito, M., and Patrizi, F. Foundations for restraining bolts: Reinforcement learning with ltlf/ldlf restraining specifications. In Proceedings of the international conference on automated planning and scheduling , volume 29, pp. 128-136, 2019.
  7. Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without a reward function. ArXiv , abs/1802.06070, 2019. · arXiv
  8. Feng, W., Zhuo, H. H., and Kambhampati, S. Extracting action sequences from texts based on deep reinforcement learning. In Lang, J. (ed.), Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden , pp. 4064-4070. ijcai.org, 2018. doi: 10.24963/ijcai.2018/565. URL https://doi.org/ 10.24963/ijcai.2018/565 . · DOI
  9. Florensa, C., Duan, Y., and Abbeel, P. Stochastic neural networks for hierarchical reinforcement learning. arXiv preprint arXiv:1704.03012 , 2017. · arXiv
  10. Geffner, H. and Bonet, B. A concise introduction to models and methods for automated planning. Synthesis Lectures on Artificial Intelligence and Machine Learning , 8(1): 1-141, 2013.
  11. Goyal, P., Niekum, S., and Mooney, R. J. Using natural language for reward shaping in reinforcement learning. arXiv preprint arXiv:1903.02020 , 2019. · arXiv
  12. Grzes, M. and Kudenko, D. Plan-based reward shaping for reinforcement learning. In 2008 4th International IEEE Conference Intelligent Systems , volume 2, pp. 1022. IEEE, 2008.
  13. Guan, L., Verma, M., Guo, S., Zhang, R., and Kambhampati, S. Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation. Advances in Neural Information Processing Systems , 34, 2021.
  14. Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning , pp. 1352-1361. PMLR, 2017.
  15. Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S. J., and Dragan, A. D. Inverse reward design. In Guyon, I., von Luxburg, U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pp. 6765-6774, 2017.
  16. Hernandez, A. O., Sreedharan, S., and Kambhampati, S. Gpt3-to-plan: Extracting plans from text using GPT-3. CoRR , abs/2106.07131, 2021. URL https://arxiv. org/abs/2106.07131 . · arXiv
  17. Icarte, R. T., Klassen, T., Valenzano, R., and McIlraith, S. Using reward machines for high-level task specification and decomposition in reinforcement learning. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pp. 21072116. PMLR, 10-15 Jul 2018.
  18. Illanes, L., Yan, X., Icarte, R. T., and McIlraith, S. A. Symbolic plans as high-level instructions for reinforcement learning. In Proceedings of the Thirtieth International Conference on Automated Planning and Scheduling, Nancy, France, October 26-30, 2020 , pp. 540-550. AAAI Press, 2020.
  19. Jothimurugan, K., Bansal, S., Bastani, O., and Alur, R. Compositional reinforcement learning from logical specifications. Advances in Neural Information Processing Systems , 34:10026-10039, 2021.
  20. Kambhampati, S. Model-lite planning for the web age masses: The challenges of planning with incomplete and evolving domain models. In Proceedings of the TwentySecond AAAI Conference on Artificial Intelligence, July 22-26, 2007, Vancouver, British Columbia, Canada , pp. 1601-1605. AAAI Press, 2007.
  21. Kambhampati, S., Ihrig, L. H., and Srivastava, B. A candidate set based analysis of subgoal interactions in conjunctive goal planning. In AIPS , pp. 125-133, 1996.
  22. Kambhampati, S., Sreedharan, S., Verma, M., Zha, Y., and Guan, L. Symbols as a lingua franca for bridging humanai chasm for explainable and advisable ai systems. In AAAI , 2022.
  23. Keyder, E., Richter, S., and Helmert, M. Sound and complete landmarks for and/or graphs. In Coelho, H., Studer, R., and Wooldridge, M. J. (eds.), ECAI 2010 - 19th European Conference on Artificial Intelligence, Lisbon, Portugal, August 16-20, 2010, Proceedings , volume 215 of Frontiers in Artificial Intelligence and Applications , pp. 335-340. IOS Press, 2010.
  24. Kokel, H., Manoharan, A., Natarajan, S., Ravindran, B., and Tadepalli, P. Reprel: Integrating relational planning and reinforcement learning for effective abstraction. In Proceedings of the International Conference on Automated Planning and Scheduling , volume 31, pp. 533-541, 2021.
  25. Kulkarni, T. D., Narasimhan, K., Saeedi, A., and Tenenbaum, J. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems , 29: 3675-3683, 2016.
  26. Kumar, S., Kumar, A., Levine, S., and Finn, C. One solution is not all you need: Few-shot extrapolation via structured maxent rl. ArXiv , abs/2010.14484, 2020. · arXiv
  27. Lee, L., Eysenbach, B., Parisotto, E., Xing, E., Levine, S., and Salakhutdinov, R. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274 , 2019. · arXiv
  28. Li, L., Walsh, T. J., and Littman, M. L. Towards a unified theory of state abstraction for mdps. In International Symposium on Artificial Intelligence and Mathematics, ISAIM 2006, Fort Lauderdale, Florida, USA, January 4-6, 2006 , 2006.
  29. Lyu, D., Yang, F., Liu, B., and Gustafson, S. Sdrl: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pp. 2970-2977, 2019.
  30. Marthi, B., Russell, S. J., and Wolfe, J. A. Angelic semantics for high-level actions. In Boddy, M. S., Fox, M., and Thi´ ebaux, S. (eds.), Proceedings of the Seventeenth International Conference on Automated Planning and Scheduling, ICAPS 2007, Providence, Rhode Island, USA, September 22-26, 2007 , pp. 232-239. AAAI, 2007.
  31. Oh, J., Guo, Y., Singh, S., and Lee, H. Self-imitation learning. In International Conference on Machine Learning , pp. 3878-3887. PMLR, 2018.
  32. Sreedharan, S., Soni, U., Verma, M., Srivastava, S., and Kambhampati, S. Bridging the gap: Providing post-hoc symbolic explanations for sequential decisionmaking problems with black box simulators. CoRR , abs/2002.01080, 2020. · arXiv
  33. Srivastava, S., Russell, S. J., and Pinto, A. Metaphysics of planning domain descriptions. In Schuurmans, D. and Wellman, M. P. (eds.), Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA , pp. 1074-1080. AAAI Press, 2016.
  34. Sutton, R. S., Precup, D., and Singh, S. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence , 112(1-2): 181-211, 1999.
  35. Tversky, A. and Kahneman, D. Probabilistic reasoning. Readings in philosophy and cognitive science , pp. 43-68, 1993.
  36. Vemula, A., Oza, Y., Bagnell, J. A., and Likhachev, M. Planning and execution using inaccurate models with provable guarantees. In Toussaint, M., Bicchi, A., and Hermans, T. (eds.), Robotics: Science and Systems XVI, Virtual Event / Corvalis, Oregon, USA, July 12-16, 2020 , 2020. doi: 10.15607/RSS.2020.XVI.001. URL https: //doi.org/10.15607/RSS.2020.XVI.001 . · DOI
  37. Vemula, A., Bagnell, J. A., and Likhachev, M. Cmax++: Leveraging experience in planning and execution using inaccurate models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pp. 6147-6155, 2021.
  38. Yang, F., Lyu, D., Liu, B., and Gustafson, S. Peorl: Integrating symbolic planning and hierarchical reinforcement learning for robust decision-making. arXiv preprint arXiv:1804.07779 , 2018. · arXiv
  39. Zhang, A., Sukhbaatar, S., Lerer, A., Szlam, A., and Fergus, R. Composable planning with attributes. In International Conference on Machine Learning , pp. 5842-5851. PMLR, 2018.
  40. Zhang, R., Torabi, F., Guan, L., Ballard, D. H., and Stone, P. Leveraging human guidance for deep reinforcement learning tasks. arXiv preprint arXiv:1909.09906 , 2019. · arXiv

Cited by

Sorted by citations

  • LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks

    Kambhampati, S., Valmeekam, K., Guan, L., Verma, M., Stechly, K., Bhambri, S., Saldyt, L., & Murthy, A. (2024) · arXiv · doi:10.48550/arxiv.2402.01817 · Cited by 11 · see citing works

    “…even in reaching a single subgoal, and the well known fact that even approximate symbolic models can help drastically improve the performance (c.f. Guan et al. (2022)), coupled with the fact that LLM’s are only too glad to dream up approximate models and goal recipes, there has been a performance…”
    BackgroundSelf-citation

Authors
Guan, L., Sreedharan, S., & Kambhampati, S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2202.02886
arXiv
2202.02886
OpenAlex
W4221153501
Cited by
6
References
0
Citations by year
2025: 2 · 2024: 2 · 2023: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Defense Advanced Research Projects Agency (W911NF19-2-0006)
  • Office of Naval Research (N00014-18, N00014-9-1-2119, N00014-18-1-2442, N00014-18-12840, FA9550, N00014)
  • Air Force Office of Scientific Research (FA9550-, FA9550‐18‐1‐0067, -16-1-)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
Keywords
skill variety, task decomposition, long-horizon tasks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review