CoChat
Sign Up
PreprintCited by 86Reinforcement Learning in RoboticsAI-based Problem Solving and PlanningMachine Learning and Algorithms

Model-based Reinforcement Learning: A Survey

Open access
Moerland, Thomas M., Broekens, Joost, Plaat, Aske, Jonker, Catholijn M.·2020·arXiv
DOI 10.48550/arxiv.2006.16712arXiv 2006.16712
Open this paper in CoChat — free accountOpen PDF

Abstract

Sequential decision making, commonly formalized as Markov Decision Process (MDP) optimization, is a important challenge in artificial intelligence. Two key approaches to this problem are reinforcement learning (RL) and planning. This paper presents a survey of the integration of both fields, better known as model-based reinforcement learning. Model-based RL has two main steps. First, we systematically cover approaches to dynamics model learning, including challenges like dealing with stochasticity, uncertainty, partial observability, and temporal abstraction. Second, we present a systematic categorization of planning-learning integration, including aspects like: where to start planning, what budgets to allocate to planning and real data collection, how to plan, and how to integrate planning in the learning and acting loop. After these two sections, we also discuss implicit model-based RL as an end-to-end alternative for model learning and planning, and we cover the potential benefits of model-based RL. Along the way, the survey also draws connections to several related RL fields, like hierarchical RL and transfer learning. Altogether, the survey presents a broad conceptual overview of the combination of planning and learning for MDP optimization.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.16712

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Reinforcement learning with world models has recently achieved notable success because it is sample-efficient and can incorporate Markov decision process techniques.

    “Reinforcement learning (RL) using world models has found significant recent successes due to its strength in sampling efficiency and ability to incorporate well-studied Markov Decision Process techniques (Moerland et al., 2020;Robine et al., 2021).”

    From Novelty Detection in Reinforcement Learning with World Models (2023) · cites this paper as Moerland et al., 2020

    The Introduction (p. 2) says model-based RL has shown great success and defines it as combining planning and learning for MDP optimization. Section 7.1 (p. 35) describes using planning to reduce real-world sample complexity. The paper does not establish that these two strengths caused its recent successes, as the citing sentence claims. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 304

  1. Abbeel, P. and Ng, A. Y. (2005). Learning first-order Markov models for control. In Advances in neural information processing systems , pages 1-8.
  2. Achiam, J., Edwards, H., Amodei, D., and Abbeel, P. (2018). Variational option discovery algorithms. arXiv preprint arXiv:1807.10299 . · arXiv
  3. Achiam, J. and Sastry, S. (2017). Surprise-based intrinsic motivation for deep reinforcement learning. arXiv preprint arXiv:1703.01732 . · arXiv
  4. Agostinelli, F., McAleer, S., Shmakov, A., and Baldi, P. (2019). Solving the Rubik's cube with deep reinforcement learning and search. Nature Machine Intelligence , 1(8):356-363.
  5. Agrawal, P., Nair, A. V., Abbeel, P., Malik, J., and Levine, S. (2016). Learning to poke by poking: Experiential learning of intuitive physics. In Advances in Neural Information Processing Systems , pages 5074-5082.
  6. Agrawal, S. and Jia, R. (2017). Posterior sampling for reinforcement learning: worst-case regret bounds. In Advances in Neural Information Processing Systems , pages 1184-1194.
  7. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Man´ e, D. (2016). Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 . · arXiv
  8. Anand, A., Racah, E., Ozair, S., Bengio, Y., Cˆ ot´ e, M.-A., and Hjelm, R. D. (2019). Unsupervised state representation learning in atari. In Advances in Neural Information Processing Systems , pages 8766-8779.
  9. Anthony, T., Nishihara, R., Moritz, P., Salimans, T., and Schulman, J. (2019). Policy Gradient Search: Online Planning and Expert Iteration without Search Trees. arXiv preprint arXiv:1904.03646 . · arXiv
  10. Anthony, T., Tian, Z., and Barber, D. (2017). Thinking fast and slow with deep learning and tree search. In Advances in Neural Information Processing Systems , pages 5360-5370.
  11. Asadi, K., Cater, E., Misra, D., and Littman, M. L. (2018). Towards a Simple Approach to Multi-step Model-based Reinforcement Learning. arXiv preprint arXiv:1811.00128 . · arXiv
  12. Asmuth, J., Li, L., Littman, M. L., Nouri, A., and Wingate, D. (2009). A Bayesian sampling approach to exploration in reinforcement learning. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence , pages 19-26. AUAI Press.
  13. ˚ Astr¨ om, K. J. and Eykhoff, P. (1971). System identification-a survey. Automatica , 7(2):123-162.
  14. Aswani, A., Gonzalez, H., Sastry, S. S., and Tomlin, C. (2013). Provably safe and robust learning-based model predictive control. Automatica , 49(5):1216-1226.
  15. Atkeson, C. G., Moore, A. W., and Schaal, S. (1997). Locally weighted learning for control. In Lazy learning , pages 75-113. Springer.
  16. Atkeson, C. G. and Santamaria, J. C. (1997). A comparison of direct and model-based reinforcement learning. In Proceedings of International Conference on Robotics and Automation , volume 4, pages 3557-3564. IEEE.
  17. Auer, P. (2002). Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422.
  18. Azar, M. G., Osband, I., and Munos, R. (2017). Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR.
  19. Babaeizadeh, M., Finn, C., Erhan, D., Campbell, R. H., and Levine, S. (2017). Stochastic variational video prediction. arXiv preprint arXiv:1710.11252 . · arXiv
  20. Bacon, P.-L., Harb, J., and Precup, D. (2017). The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence .
  21. Bagnell, J. A. and Schneider, J. G. (2001). Autonomous helicopter control using reinforcement learning policy search methods. In Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No. 01CH37164) , volume 2, pages 16151620. IEEE.
  22. Baranes, A. and Oudeyer, P.-Y. (2009). R-iac: Robust intrinsically motivated exploration and active learning. IEEE Transactions on Autonomous Mental Development , 1(3):155169.
  23. Baranes, A. and Oudeyer, P.-Y. (2013). Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems , 61(1):49-73.
  24. Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., van Hasselt, H. P., and Silver, D. (2017). Successor features for transfer in reinforcement learning. In Advances in neural information processing systems , pages 4055-4065.
  25. Barto, A. G., Bradtke, S. J., and Singh, S. P. (1995). Learning to act using real-time dynamic programming. Artificial intelligence , 72(1-2):81-138.
  26. Barto, A. G. and Mahadevan, S. (2003). Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems , 13(1-2):41-77.
  27. Battaglia, P., Pascanu, R., Lai, M., Rezende, D. J., et al. (2016). Interaction networks for learning about objects, relations and physics. In Advances in neural information processing systems , pages 4502-4510.
  28. Battaglia, P. W., Hamrick, J. B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., et al. (2018). Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 . · arXiv
  29. Baxter, J., Tridgell, A., and Weaver, L. (1999). TDLeaf (lambda): Combining temporal difference learning with game-tree search. arXiv preprint cs/9901001 . · arXiv
  30. Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. (2016). Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems , pages 1471-1479.
  31. Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. (2013). The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research , 47:253-279.
  32. Bellman, R. (1966). Dynamic programming. Science , 153(3731):34-37.
  33. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009). Curriculum learning. In Proceedings of the 26th annual international conference on machine learning , pages 41-48. ACM.
  34. Berkenkamp, F., Turchetta, M., Schoellig, A., and Krause, A. (2017). Safe model-based reinforcement learning with stability guarantees. In Advances in neural information processing systems , pages 908-918.
  35. Blundell, C., Uria, B., Pritzel, A., Li, Y., Ruderman, A., Leibo, J. Z., Rae, J., Wierstra, D., and Hassabis, D. (2016). Model-free episodic control. arXiv preprint arXiv:1606.04460 . · arXiv
  36. Boone, G. (1997). Efficient reinforcement learning: Model-based acrobot control. In Proceedings of International Conference on Robotics and Automation , volume 1, pages 229-234. IEEE.
  37. Brafman, R. I. and Tennenholtz, M. (2002). R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research , 3(Oct):213231.
  38. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). OpenAI Gym. arXiv preprint arXiv:1606.01540 . · arXiv
  39. Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., and Colton, S. (2012). A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games , 4(1):1-43.
  40. Brunskill, E. and Li, L. (2014). Pac-inspired option discovery in lifelong reinforcement learning. In International conference on machine learning , pages 316-324.
  41. Buckman, J., Hafner, D., Tucker, G., Brevdo, E., and Lee, H. (2018). Sample-efficient reinforcement learning with stochastic ensemble value expansion. In Advances in Neural Information Processing Systems , pages 8224-8234.
  42. Buesing, L., Weber, T., Racaniere, S., Eslami, S., Rezende, D., Reichert, D. P., Viola, F., Besse, F., Gregor, K., Hassabis, D., et al. (2018). Learning and querying fast generative models for reinforcement learning. arXiv preprint arXiv:1802.03006 . · arXiv
  43. Caruana, R. (1997). Multitask learning. Machine learning , 28(1):41-75.
  44. Castro, P. S. and Precup, D. (2007). Using Linear Programming for Bayesian Exploration in Markov Decision Processes. In IJCAI , volume 24372442.
  45. Chang, M. B., Ullman, T., Torralba, A., and Tenenbaum, J. B. (2016). A compositional object-based approach to learning physical dynamics. arXiv preprint arXiv:1612.00341 . · arXiv
  46. Chentanez, N., Barto, A. G., and Singh, S. P. (2005). Intrinsically motivated reinforcement learning. In Advances in neural information processing systems , pages 1281-1288.
  47. Chiappa, S., Racaniere, S., Wierstra, D., and Mohamed, S. (2017). Recurrent environment simulators. arXiv preprint arXiv:1704.02254 . · arXiv
  48. Choi, J., Guo, Y., Moczulski, M., Oh, J., Wu, N., Norouzi, M., and Lee, H. (2018). Contingency-aware exploration in reinforcement learning. arXiv preprint arXiv:1811.01483 . · arXiv
  49. Chrisman, L. (1992). Reinforcement learning with perceptual aliasing: The perceptual distinctions approach. In AAAI , volume 1992, pages 183-188. Citeseer.
  50. Christiano, P., Shah, Z., Mordatch, I., Schneider, J., Blackwell, T., Tobin, J., Abbeel, P., and Zaremba, W. (2016). Transfer from simulation to real world through learning deep inverse dynamics model. arXiv preprint arXiv:1610.03518 . · arXiv
  51. Chua, K., Calandra, R., McAllister, R., and Levine, S. (2018). Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems , pages 4754-4765.
  52. Clavera, I., Rothfuss, J., Schulman, J., Fujita, Y., Asfour, T., and Abbeel, P. (2018). ModelBased Reinforcement Learning via Meta-Policy Optimization. In Conference on Robot Learning , pages 617-629.
  53. Corneil, D., Gerstner, W., and Brea, J. (2018). Efficient model-based deep reinforcement learning with variational state tabulation. arXiv preprint arXiv:1802.04325 . · arXiv
  54. Da Silva, B. C., Basso, E. W., Bazzan, A. L., and Engel, P. M. (2006). Dealing with nonstationary environments using context detection. In Proceedings of the 23rd international conference on Machine learning , pages 217-224. ACM.
  55. Daniel, C., Van Hoof, H., Peters, J., and Neumann, G. (2016). Probabilistic inference for determining options in reinforcement learning. Machine Learning , 104(2-3):337-357.
  56. Dann, C. and Brunskill, E. (2015). Sample complexity of episodic fixed-horizon reinforcement learning. In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 2 , pages 2818-2826.
  57. Dayan, P. (1993). Improving generalization for temporal difference learning: The successor representation. Neural Computation , 5(4):613-624.
  58. Dayan, P. and Hinton, G. E. (1993). Feudal reinforcement learning. In Advances in neural information processing systems , pages 271-278.
  59. de Avila Belbute-Peres, F., Smith, K., Allen, K., Tenenbaum, J., and Kolter, J. Z. (2018). End-to-end differentiable physics for learning and control. In Advances in Neural Information Processing Systems , pages 7178-7189.
  60. Dearden, R., Friedman, N., and Andre, D. (1999). Model based Bayesian exploration. In Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence , pages 150-159. Morgan Kaufmann Publishers Inc.
  61. Dearden, R., Friedman, N., and Russell, S. (1998). Bayesian Q-learning. In AAAI/IAAI , pages 761-768.
  62. Deisenroth, M. and Rasmussen, C. E. (2011). PILCO: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11) , pages 465-472.
  63. Depeweg, S., Hern´ andez-Lobato, J. M., Doshi-Velez, F., and Udluft, S. (2016). Learning and policy search in stochastic dynamical systems with bayesian neural networks. arXiv preprint arXiv:1605.07127 . · arXiv
  64. Der Kiureghian, A. and Ditlevsen, O. (2009). Aleatory or epistemic? Does it matter? Structural Safety , 31(2):105-112.
  65. Dilokthanakul, N., Kaplanis, C., Pawlowski, N., and Shanahan, M. (2019). Feature control as intrinsic motivation for hierarchical reinforcement learning. IEEE transactions on neural networks and learning systems .
  66. Diuk, C., Cohen, A., and Littman, M. L. (2008). An object-oriented representation for efficient reinforcement learning. In Proceedings of the 25th international conference on Machine learning , pages 240-247. ACM.
  67. Doll, B. B., Simon, D. A., and Daw, N. D. (2012). The ubiquity of model-based reinforcement learning. Current opinion in neurobiology , 22(6):1075-1081.
  68. Doya, K., Samejima, K., Katagiri, K.-i., and Kawato, M. (2002). Multiple model-based reinforcement learning. Neural computation , 14(6):1347-1369.
  69. Duff, M. O. and Barto, A. (2002). Optimal Learning: Computational procedures for Bayesadaptive Markov decision processes . PhD thesis, University of Massachusetts at Amherst.
  70. Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., and Clune, J. (2019). Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995 . · arXiv
  71. Edwards, A. D., Downs, L., and Davidson, J. C. (2018). Forward-backward reinforcement learning. arXiv preprint arXiv:1803.10227 . · arXiv
  72. Efroni, Y., Dalal, G., Scherrer, B., and Mannor, S. (2018). Beyond the One-Step Greedy Approach in Reinforcement Learning. In International Conference on Machine Learning , pages 1386-1395.
  73. Efroni, Y., Ghavamzadeh, M., and Mannor, S. (2019a). Multi-Step Greedy and Approximate Real Time Dynamic Programming. arXiv preprint arXiv:1909.04236 . · arXiv
  74. Efroni, Y., Merlis, N., Ghavamzadeh, M., and Mannor, S. (2019b). Tight Regret Bounds for Model-Based Reinforcement Learning with Greedy Policies. arXiv preprint arXiv:1905.11527 . · arXiv
  75. El Hihi, S. and Bengio, Y. (1996). Hierarchical recurrent neural networks for long-term dependencies. In Advances in neural information processing systems , pages 493-499.
  76. Evans, J. S. B. (1984). Heuristic and analytic processes in reasoning. British Journal of Psychology , 75(4):451-468.
  77. Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. (2019a). Diversity is All You Need: Learning Skills without a Reward Function. In International Conference on Learning Representations .
  78. Eysenbach, B., Salakhutdinov, R. R., and Levine, S. (2019b). Search on the Replay Buffer: Bridging Planning and Reinforcement Learning. Advances in Neural Information Processing Systems , 32.
  79. Fairbank, M. and Alonso, E. (2012). Value-gradient learning. In The 2012 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE.
  80. Farquhar, G., Rockt¨ aschel, T., Igl, M., and Whiteson, S. (2018). Treeqn and atreec: Differentiable tree planning for deep reinforcement learning. International Conference on Learning Representations.
  81. Finn, C., Goodfellow, I., and Levine, S. (2016). Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems , pages 64-72.
  82. Florensa, C., Duan, Y., and Abbeel, P. (2017). Stochastic neural networks for hierarchical reinforcement learning. arXiv preprint arXiv:1704.03012 . · arXiv
  83. Florensa, C., Held, D., Geng, X., and Abbeel, P. (2018). Automatic Goal Generation for Reinforcement Learning Agents. In International Conference on Machine Learning , pages 1514-1523.
  84. Fortunato, M., Tan, M., Faulkner, R., Hansen, S., Badia, A. P., Buttimore, G., Deck, C., Leibo, J. Z., and Blundell, C. (2019). Generalization of reinforcement learners with working and episodic memory. arXiv preprint arXiv:1910.13406 . · arXiv
  85. Fox, R., Krishnan, S., Stoica, I., and Goldberg, K. (2017). Multi-level discovery of deep options. arXiv preprint arXiv:1703.08294 . · arXiv
  86. Fox, R., Moshkovitz, M., and Tishby, N. (2016). Principled option learning in Markov decision processes. arXiv preprint arXiv:1609.05524 . · arXiv
  87. Fraccaro, M., Kamronn, S., Paquet, U., and Winther, O. (2017). A disentangled recognition and nonlinear dynamics model for unsupervised learning. In Advances in Neural Information Processing Systems , pages 3601-3610.
  88. Fragkiadaki, K., Agrawal, P., Levine, S., and Malik, J. (2015). Learning visual predictive models of physics for playing billiards. arXiv preprint arXiv:1511.07404 . · arXiv
  89. Fran¸ cois-Lavet, V., Bengio, Y., Precup, D., and Pineau, J. (2019). Combined reinforcement learning via abstract representations. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 3582-3589.
  90. Frans, K., Ho, J., Chen, X., Abbeel, P., and Schulman, J. (2018). Meta learning shared hierarchies. In International Conference on Learning Representations .
  91. Fr¨ ohlich, F., Theis, F. J., and Hasenauer, J. (2014). Uncertainty analysis for non-identifiable dynamical systems: Profile likelihoods, bootstrapping and more. In International Conference on Computational Methods in Systems Biology , pages 61-72. Springer.
  92. Gal, Y., McAllister, R., and Rasmussen, C. E. (2016). Improving PILCO with Bayesian neural network dynamics models. In Data-Efficient Machine Learning workshop, ICML , volume 4.
  93. Gemici, M., Hung, C.-C., Santoro, A., Wayne, G., Mohamed, S., Rezende, D. J., Amos, D., and Lillicrap, T. (2017). Generative temporal models with memory. arXiv preprint arXiv:1702.04649 . · arXiv
  94. Gershman, S. J. and Daw, N. D. (2017). Reinforcement learning and episodic memory in humans and animals: an integrative framework. Annual review of psychology , 68:101-128.
  95. Ghahramani, Z. and Roweis, S. T. (1999). Learning nonlinear dynamical systems using an EM algorithm. In Advances in neural information processing systems , pages 431-437.
  96. Ghavamzadeh, M., Mannor, S., Pineau, J., Tamar, A., et al. (2015). Bayesian reinforcement learning: A survey. Foundations and Trends ® in Machine Learning , 8(5-6):359-483.
  97. Ghosh, D., Gupta, A., and Levine, S. (2018). Learning Actionable Representations with Goal-Conditioned Policies. arXiv preprint arXiv:1811.07819 . · arXiv
  98. Goel, S. and Huber, M. (2003). Subgoal discovery for hierarchical reinforcement learning using learned policies. In FLAIRS conference , pages 346-350.
  99. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep learning . MIT press.
  100. Gopalan, A. and Mannor, S. (2015). Thompson sampling for learning parameterized markov decision processes. In Conference on Learning Theory , pages 861-898. PMLR.
  101. Graves, A., Wayne, G., and Danihelka, I. (2014). Neural turing machines. arXiv preprint arXiv:1410.5401 . · arXiv
  102. Gregor, K., Rezende, D. J., and Wierstra, D. (2016). Variational intrinsic control. arXiv preprint arXiv:1611.07507 . · arXiv
  103. Grimm, C., Barreto, A., Singh, S., and Silver, D. (2020). The Value Equivalence Principle for Model-Based Reinforcement Learning. Advances in Neural Information Processing Systems .
  104. Gu, S., Lillicrap, T., Sutskever, I., and Levine, S. (2016). Continuous deep q-learning with model-based acceleration. In International Conference on Machine Learning , pages 28292838.
  105. Guestrin, C., Koller, D., Gearhart, C., and Kanodia, N. (2003). Generalizing plans to new environments in relational MDPs. In Proceedings of the 18th international joint conference on Artificial intelligence , pages 1003-1010. Morgan Kaufmann Publishers Inc.
  106. Guez, A., Mirza, M., Gregor, K., Kabra, R., Racaniere, S., Weber, T., Raposo, D., Santoro, A., Orseau, L., Eccles, T., et al. (2019). An Investigation of Model-Free Planning. In International Conference on Machine Learning , pages 2464-2473.
  107. Guez, A., Silver, D., and Dayan, P. (2012). Efficient Bayes-adaptive reinforcement learning using sample-based search. In Advances in neural information processing systems , pages 1025-1033.
  108. Guez, A., Weber, T., Antonoglou, I., Simonyan, K., Vinyals, O., Wierstra, D., Munos, R., and Silver, D. (2018). Learning to search with MCTSnets. arXiv preprint arXiv:1802.04697 . · arXiv
  109. Guo, X., Singh, S., Lee, H., Lewis, R. L., and Wang, X. (2014). Deep learning for real-time Atari game play using offline Monte-Carlo tree search planning. In Advances in neural information processing systems , pages 3338-3346.
  110. Ha, D. and Schmidhuber, J. (2018). Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems , pages 2450-2462.
  111. Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. (2019a). Dream to Control: Learning Behaviors by Latent Imagination. In International Conference on Learning Representations .
  112. Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. (2019b). Learning latent dynamics for planning from pixels. In International Conference on Machine Learning , pages 2555-2565. PMLR.
  113. Hamidi, M., Tadepalli, P., Goetschalckx, R., and Fern, A. (2015). Active imitation learning of hierarchical policies. In Twenty-Fourth International Joint Conference on Artificial Intelligence .
  114. Hamrick, J. B., Ballard, A. J., Pascanu, R., Vinyals, O., Heess, N., and Battaglia, P. W. (2017). Metacontrol for adaptive imagination-based optimization. arXiv preprint arXiv:1705.02670 . · arXiv
  115. Hamrick, J. B., Bapst, V., Sanchez-Gonzalez, A., Pfaff, T., Weber, T., Buesing, L., and Battaglia, P. W. (2020). Combining q-learning and search with amortized value estimates. International Conference on Learning Representations (ICLR) .
  116. Hausman, K., Springenberg, J. T., Wang, Z., Heess, N., and Riedmiller, M. (2018). Learning an Embedding Space for Transferable Robot Skills. In International Conference on Learning Representations .
  117. Heess, N., Wayne, G., Silver, D., Lillicrap, T., Erez, T., and Tassa, Y. (2015). Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems , pages 2944-2952.
  118. Heess, N., Wayne, G., Tassa, Y., Lillicrap, T., Riedmiller, M., and Silver, D. (2016). Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv:1610.05182 . · arXiv
  119. Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. (2018). Deep reinforcement learning that matters. In Thirty-Second AAAI Conference on Artificial Intelligence .
  120. Hengst, B. (2017). Hierarchical reinforcement learning. Encyclopedia of Machine Learning and Data Mining , pages 611-619.
  121. Hester, T. and Stone, P. (2012a). Intrinsically motivated model learning for a developing curious agent. In 2012 IEEE international conference on development and learning and epigenetic robotics (ICDL) , pages 1-6. IEEE.
  122. Hester, T. and Stone, P. (2012b). Learning and using models. In Reinforcement learning , pages 111-141. Springer.
  123. Hester, T. and Stone, P. (2013). TEXPLORE: real-time sample-efficient reinforcement learning for robots. Machine learning , 90(3):385-429.
  124. Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation , 9(8):1735-1780.
  125. Holland, G. Z., Talvitie, E. J., and Bowling, M. (2018). The effect of planning shape on dyna-style planning in high-dimensional state spaces. arXiv preprint arXiv:1806.01825 . · arXiv
  126. Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F., and Abbeel, P. (2016). Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems , pages 1109-1117.
  127. Hu, H., Ye, J., Zhu, G., Ren, Z., and Zhang, C. (2021). Generalizable episodic memory for deep reinforcement learning. arXiv preprint arXiv:2103.06469 . · arXiv
  128. Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K. (2016). Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397 . · arXiv
  129. Jaksch, T., Ortner, R., and Auer, P. (2010). Near-optimal Regret Bounds for Reinforcement Learning. Journal of Machine Learning Research , 11(4).
  130. Janner, M., Fu, J., Zhang, M., and Levine, S. (2019). When to trust your model: Modelbased policy optimization. In Advances in Neural Information Processing Systems , pages 12519-12530.
  131. Jaulmes, R., Pineau, J., and Precup, D. (2005). Learning in non-stationary partially observable Markov decision processes. In ECML Workshop on Reinforcement Learning in non-stationary environments , volume 25, pages 26-32.
  132. Jayaraman, D., Ebert, F., Efros, A. A., and Levine, S. (2018). Time-agnostic prediction: Predicting predictable video frames. arXiv preprint arXiv:1808.07784 . · arXiv
  133. Jiang, D., Ekwedike, E., and Liu, H. (2018). Feedback-Based Tree Search for Reinforcement Learning. In International Conference on Machine Learning , pages 2289-2298.
  134. Jin, C., Allen-Zhu, Z., Bubeck, S., and Jordan, M. I. (2018). Is Q-learning provably efficient? In Proceedings of the 32nd International Conference on Neural Information Processing Systems , pages 4868-4878.
  135. Jong, N. K. and Stone, P. (2007). Model-based function approximation in reinforcement learning. In Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems , page 95. ACM.
  136. Jonschkowski, R. and Brock, O. (2015). Learning state representations with robotic priors. Autonomous Robots , 39(3):407-428.
  137. Jordan, M. I. and Rumelhart, D. E. (1992). Forward models: Supervised learning with a distal teacher. Cognitive science , 16(3):307-354.
  138. Kakade, S. and Langford, J. (2002). Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning . Citeseer.
  139. Kakade, S. M. et al. (2003). On the sample complexity of reinforcement learning . PhD thesis, University of London London, England.
  140. Kalchbrenner, N., van den Oord, A., Simonyan, K., Danihelka, I., Vinyals, O., Graves, A., and Kavukcuoglu, K. (2017). Video pixel networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1771-1779. JMLR. org.
  141. Kalweit, G. and Boedecker, J. (2017). Uncertainty-driven imagination for continuous deep reinforcement learning. In Conference on Robot Learning , pages 195-206.
  142. Kamthe, S. and Deisenroth, M. P. (2017). Data-efficient reinforcement learning with probabilistic model predictive control. arXiv preprint arXiv:1706.06491 . · arXiv
  143. Kansky, K., Silver, T., M´ ely, D. A., Eldawy, M., L´ azaro-Gredilla, M., Lou, X., Dorfman, N., Sidor, S., Phoenix, S., and George, D. (2017). Schema networks: Zero-shot transfer with a generative causal model of intuitive physics. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1809-1818. JMLR. org.
  144. Karl, M., Soelch, M., Bayer, J., and van der Smagt, P. (2016). Deep variational bayes filters: Unsupervised learning of state space models from raw data. arXiv preprint arXiv:1605.06432 . · arXiv
  145. Ke, N. R., Singh, A., Touati, A., Goyal, A., Bengio, Y., Parikh, D., and Batra, D. (2019). Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future. arXiv preprint arXiv:1903.01599 . · arXiv
  146. Khansari-Zadeh, S. M. and Billard, A. (2011). Learning stable nonlinear dynamical systems with gaussian mixture models. IEEE Transactions on Robotics , 27(5):943-957.
  147. Kipf, T., van der Pol, E., and Welling, M. (2020). Contrastive Learning of Structured World Models. In International Conference on Learning Representations .
  148. Kober, J., Bagnell, J. A., and Peters, J. (2013). Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32(11):1238-1274.
  149. Kolter, J. Z. and Ng, A. Y. (2009). Near-Bayesian exploration in polynomial time. In Proceedings of the 26th Annual International Conference on Machine Learning , pages 513-520. ACM.
  150. Konidaris, G. and Barto, A. G. (2007). Building Portable Options: Skill Transfer in Reinforcement Learning. In IJCAI , volume 7, pages 895-900.
  151. Konidaris, G., Kuindersma, S., Grupen, R., and Barto, A. (2012). Robot learning from demonstration by constructing skill trees. The International Journal of Robotics Research , 31(3):360-375.
  152. Konidaris, G. D. (2006). A framework for transfer in reinforcement learning. In ICML-06 Workshop on Structural Knowledge Transfer for Machine Learning .
  153. Korf, R. E. (1990). Real-time heuristic search. Artificial intelligence , 42(2-3):189-211.
  154. Krishnan, R. G., Shalit, U., and Sontag, D. A. (2015). Deep Kalman Filters. ArXiv , abs/1511.05121. · arXiv
  155. Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pages 1097-1105.
  156. Kulkarni, T. D., Narasimhan, K., Saeedi, A., and Tenenbaum, J. (2016). Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems , pages 3675-3683.
  157. Kurutach, T., Tamar, A., Yang, G., Russell, S. J., and Abbeel, P. (2018). Learning plannable representations with causal infogan. In Advances in Neural Information Processing Systems , pages 8733-8744.
  158. Lai, M. (2015). Giraffe: Using deep reinforcement learning to play chess. arXiv preprint arXiv:1509.01549 . · arXiv
  159. Lai, T. L. and Robbins, H. (1985). Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22.
  160. Lakshminarayanan, A. S., Krishnamurthy, R., Kumar, P., and Ravindran, B. (2016). Option discovery in hierarchical reinforcement learning using spatio-temporal clustering. arXiv preprint arXiv:1605.05359 . · arXiv
  161. Lange, S., Gabel, T., and Riedmiller, M. (2012). Batch reinforcement learning. In Reinforcement learning , pages 45-73. Springer.
  162. Laversanne-Finot, A., Pere, A., and Oudeyer, P.-Y. (2018). Curiosity Driven Exploration of Learned Disentangled Goal Spaces. In Conference on Robot Learning , pages 487-504.
  163. Lazaric, A. (2012). Transfer in reinforcement learning: a framework and a survey. In Reinforcement Learning , pages 143-173. Springer.
  164. Levine, S. and Abbeel, P. (2014). Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems , pages 1071-1079.
  165. Levine, S. and Koltun, V. (2013). Guided policy search. In International Conference on Machine Learning , pages 1-9.
  166. Levy, A., Platt, R., and Saenko, K. (2019). Hierarchical Reinforcement Learning with Hindsight. In International Conference on Learning Representations .
  167. Lin, L.-J. (1992). Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning , 8(3-4):293-321.
  168. Lin, Z., Zhao, T., Yang, G., and Zhang, L. (2018). Episodic memory deep Q-networks. arXiv preprint arXiv:1805.07603 . · arXiv
  169. Ljung, L. (2001). System identification. Wiley Encyclopedia of Electrical and Electronics Engineering .
  170. Lopes, M., Lang, T., Toussaint, M., and Oudeyer, P.-Y. (2012). Exploration in model-based reinforcement learning by empirically estimating learning progress. In Advances in neural information processing systems , pages 206-214.
  171. Lowrey, K., Rajeswaran, A., Kakade, S., Todorov, E., and Mordatch, I. (2018). Plan online, learn offline: Efficient learning and exploration via model-based control. arXiv preprint arXiv:1811.01848 . · arXiv
  172. Lu, K., Mordatch, I., and Abbeel, P. (2019). Adaptive Online Planning for Continual Lifelong Learning. arXiv preprint arXiv:1912.01188 . · arXiv
  173. Machado, M. C., Bellemare, M. G., and Bowling, M. (2017). A laplacian framework for option discovery in reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 2295-2304. JMLR. org.
  174. Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., and Bowling, M. (2018). Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research , 61:523-562.
  175. Mahadevan, S. (2009). Learning Representation and Control in Markov Decision Processes: New Frontiers. Foundations and Trends ® in Machine Learning , 1(4):403-565.
  176. Mann, T. and Mannor, S. (2014). Scaling up approximate value iteration with options: Better policies with fewer iterations. In International conference on machine learning , pages 127-135.
  177. Mannor, S., Menache, I., Hoze, A., and Klein, U. (2004). Dynamic abstraction in reinforcement learning via clustering. In Proceedings of the twenty-first international conference on Machine learning , page 71. ACM.
  178. Matiisen, T., Oliver, A., Cohen, T., and Schulman, J. (2017). Teacher-student curriculum learning. arXiv preprint arXiv:1707.00183 . · arXiv
  179. Menache, I., Mannor, S., and Shimkin, N. (2002). Q-cut-dynamic discovery of sub-goals in reinforcement learning. In European Conference on Machine Learning , pages 295-306. Springer.
  180. Mishra, N., Abbeel, P., and Mordatch, I. (2017). Prediction and control with temporal segment models. In Proceedings of the 34th International Conference on Machine LearningVolume 70 , pages 2459-2468. JMLR. org.
  181. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep reinforcement learning. Nature , 518(7540):529.
  182. Moerland, T. M., Broekens, J., and Jonker, C. M. (2017a). Efficient exploration with double uncertain value networks. Deep Reinforcement Learning Symposium, 31st Conference on Neural Information Processing Systems (NIPS) .
  183. Moerland, T. M., Broekens, J., and Jonker, C. M. (2017b). Learning Multimodal Transition Dynamics for Model-Based Reinforcement Learning. Scaling Up Reinforcement Learning (SURL) Workshop, European Conference on Machine Learning (ECML) .
  184. Moerland, T. M., Broekens, J., and Jonker, C. M. (2018a). Emotion in reinforcement learning agents and robots: a survey. Machine Learning , 107(2):443-480.
  185. Moerland, T. M., Broekens, J., and Jonker, C. M. (2020a). A Framework for Reinforcement Learning and Planning. arXiv preprint arXiv:2006.15009 . · arXiv
  186. Moerland, T. M., Broekens, J., Plaat, A., and Jonker, C. M. (2018b). A0C: Alpha zero in continuous action space. Planning and Learning Workshop, 35th International Conference on Machine Learning (ICML) .
  187. Moerland, T. M., Deichler, A., Baldi, S., Broekens, J., and Jonker, C. M. (2020b). Think Too Fast Nor Too Slow: The Computational Trade-off Between Planning And Reinforcement Learning. arXiv preprint arXiv:2005.07404 . · arXiv
  188. Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M., Daw, N. D., and Gershman, S. J. (2017). The successor representation in human reinforcement learning. Nature Human Behaviour , 1(9):680.
  189. Moore, A. W. and Atkeson, C. G. (1993). Prioritized sweeping: Reinforcement learning with less data and less time. Machine learning , 13(1):103-130.
  190. M¨ uller, K.-R., Smola, A. J., R¨ atsch, G., Sch¨ olkopf, B., Kohlmorgen, J., and Vapnik, V. (1997). Predicting time series with support vector machines. In International Conference on Artificial Neural Networks , pages 999-1004. Springer.
  191. Munos, R. (2003). Error bounds for approximate policy iteration. In ICML , volume 3, pages 560-567.
  192. Nachum, O., Gu, S. S., Lee, H., and Levine, S. (2018). Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems , pages 3303-3313.
  193. Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., and Finn, C. (2018a). Learning to Adapt in Dynamic, Real-World Environments through MetaReinforcement Learning. In International Conference on Learning Representations .
  194. Nagabandi, A., Finn, C., and Levine, S. (2018b). Deep online learning via meta-learning: Continual adaptation for model-based RL. arXiv preprint arXiv:1812.07671 . · arXiv
  195. Nagabandi, A., Kahn, G., Fearing, R. S., and Levine, S. (2018c). Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 7559-7566. IEEE.
  196. Narendra, K. S. and Parthasarathy, K. (1990). Identification and control of dynamical systems using neural networks. IEEE Transactions on neural networks , 1(1):4-27.
  197. Neitz, A., Parascandolo, G., Bauer, S., and Sch¨ olkopf, B. (2018). Adaptive skip intervals: Temporal abstraction for recurrent dynamical models. In Advances in Neural Information Processing Systems , pages 9816-9826.
  198. Nguyen-Tuong, D. and Peters, J. (2011). Model learning for robot control: a survey. Cognitive processing , 12(4):319-340.
  199. Nouri, A. and Littman, M. L. (2010). Dimension reduction and its application to modelbased exploration in continuous spaces. Machine Learning , 81(1):85-98.
  200. Oh, J., Guo, X., Lee, H., Lewis, R. L., and Singh, S. (2015). Action-conditional video prediction using deep networks in atari games. In Advances in neural information processing systems , pages 2863-2871.

200 of 304 shown

Cited by

Sorted by citations

Authors
Moerland, T. M., Broekens, J., Plaat, A., & Jonker, C. M.
Affiliations
Leiden University (NL)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2006.16712
arXiv
2006.16712
OpenAlex
W4287750003
Cited by
86
References
0
Citations by year
2026: 1 · 2025: 5 · 2024: 4 · 2023: 25 · 2022: 35 · 2021: 15 · 2020: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
model-based reinforcement learning, Markov decision process, sequential decision making, temporal abstraction, partial observability, uncertainty, hierarchical reinforcement learning, transfer learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review