Generalized Hidden Parameter MDPs:Transferable Model-Based RL in a Handful of Trials
Open accessAbstract
There is broad interest in creating RL agents that can solve many (related) tasks and adapt to new tasks and environments after initial training. Model-based RL leverages learned surrogate models that describe dynamics and rewards of individual tasks, such that planning in a good surrogate can lead to good control of the true system. Rather than solving each task individually from scratch, hierarchical models can exploit the fact that tasks are often related by (unobserved) causal factors of variation in order to achieve efficient generalization, as in learning how the mass of an item affects the force required to lift it can generalize to previously unobserved masses. We propose Generalized Hidden Parameter MDPs (GHP-MDPs) that describe a family of MDPs where both dynamics and reward can change as a function of hidden parameters that vary across tasks. The GHP-MDP augments model-based RL with latent variables that capture these hidden parameters, facilitating transfer across tasks. We also explore a variant of the model that incorporates explicit latent structure mirroring the causal factors of variation across tasks (for instance: agent properties, environmental factors, and goals). We experimentally demonstrate state-of-the-art performance and sample-efficiency on a new challenging MuJoCo task using reward and dynamics latent spaces, while beating a previous state-of-the-art baseline with > 10× less data. Using test-time inference of the latent variables, our approach generalizes in a single episode to novel combinations of dynamics and reward, and to novel rewards.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Perez et al. (2020) build on Hi-MDPs to present Generalised Hidden-parameter MDPs (GHP-MDPs), where hidden parameters also control the reward function in addition to the dynamics.
“Perez et al. (2020) builds on Hi-MDPs to present Generalised Hidden-parameter MDPs (GHP-MDPs), where the hidden parameters now also control the reward function in addition to the dynamics.”
From A Survey of Zero-shot Generalisation in Deep Reinforcement Learning (2023) · cites this paper as Perez et al. (2020)
Cites
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512 · see citing works
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767 · see citing works
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666 · see citing works
A Tutorial on the Cross-Entropy Method
Boer, P., Kroese, D. P., Mannor, S., & Rubinstein, R. (2005) · Annals of Operations Research · doi:10.1007/s10479-005-5724-z · Cited by 3,348 · see citing works
Optimization as a Model for Few-Shot Learning
Ravi, S., & Larochelle, H. (2017) · International Conference on Learning Representations · Cited by 2,766 · see citing works
Bellman, R. (1957) · Indiana University Mathematics Journal · doi:10.1512/iumj.1957.6.56038 · Cited by 2,408 · see citing works
Transfer Learning for Reinforcement Learning Domains: A Survey
Taylor, M. E., & Stone, P. (2009) · doi:10.5555/1577069.1755839 · Cited by 1,521 · see citing works
Camacho, E. F., & Bordons, C. (1999) · Advanced textbooks in control and signal processing · doi:10.1007/978-1-4471-3398-8 · Cited by 1,155 · see citing works
A Survey on Policy Search for Robotics
Deisenroth, M. P. (2011) · Foundations and Trends in Robotics · doi:10.1561/2300000021 · Cited by 815 · see citing works
Learning Latent Dynamics for Planning from Pixels
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., & Davidson, J. (2018) · arXiv · doi:10.48550/arxiv.1811.04551 · Cited by 708 · see citing works
Black Box Variational InferencePreprint
Ranganath, R., Gerrish, S., & Blei, D. M. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.0118 · Cited by 646 · see citing works
Deep reinforcement learning in a handful of trials using probabilistic dynamics models
Chua, K., Calandra, R., McAllister, R., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1805.12114 · Cited by 525 · see citing works
RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I., & Abbeel, P. (2016) · arXiv · doi:10.48550/arxiv.1611.02779 · Cited by 503 · see citing works
Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context VariablesPreprint
Rakelly, K., Zhou, A., Quillen, D., Finn, C., & Levine, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.08254 · Cited by 371 · see citing works
Successor Features for Transfer in Reinforcement LearningPreprint
Barreto, A. D. M. S., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., Silver, D., & van Hasselt, H. P. (2016) · arXiv (Cornell University) · doi:10.13140/rg.2.2.35439.18085 · Cited by 349 · see citing works
Transfer in Reinforcement Learning: A Framework and a Survey
Lazaric, A. (2012) · Adaptation, learning, and optimization · doi:10.1007/978-3-642-27645-3_5 · Cited by 270 · see citing works
Continuous Adaptation via Meta-Learning in Nonstationary and Competitive EnvironmentsPreprint
Al-Shedivat, M., Bansal, T., Burda, Y., Sutskever, I., Mordatch, I., & Abbeel, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.03641 · Cited by 263 · see citing works
Benchmarking Model-Based Reinforcement Learning
Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Langlois, E., Zhang, S., Zhang, G., Abbeel, P., & Ba, J. (2019) · arXiv · doi:10.48550/arxiv.1907.02057 · Cited by 226 · see citing works
Stochastic Gradient VB and the Variational Auto-EncoderPreprint
Kingma, D. P., & Welling, M. (2013) · arXiv (Cornell University) · Cited by 223 · see citing works
Learning an Embedding Space for Transferable Robot Skills
Hausman, K., Springenberg, J. T., Wang, Z., Heess, N., & Riedmiller, M. (2018) · International Conference on Learning Representations · Cited by 189 · see citing works
Doshi-Velez, F., & Konidaris, G. (2013) · International Joint Conference on Artificial Intelligence · doi:10.48550/arxiv.1308.3513 · Cited by 149 · see citing works
Model-Based Reinforcement Learning via Meta-Policy OptimizationPreprint
Clavera, I., Rothfuss, J., Schulman, J., Fujita, Y., Asfour, T., & Abbeel, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.05214 · Cited by 148 · see citing works
One-shot learning of manipulation skills with online dynamics adaptation and neural network priors
Fu, J., Levine, S., & Abbeel, P. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759592 · Cited by 126 · see citing works
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLPreprint
Nagabandi, A., Finn, C., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.07671 · Cited by 120 · see citing works
DeepMDP: Learning Continuous Latent Space Models for Representation LearningPreprint
Gelada, C., Kumar, S., Buckman, J., Nachum, O., & Bellemare, M. G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02736 · Cited by 118 · see citing works
Meta Reinforcement Learning with Latent Variable Gaussian Processes
Sæmundsson, S., Hofmann, K., & Deisenroth, M. P. (2018) · arXiv · doi:10.48550/arxiv.1803.07551 · Cited by 106 · see citing works
Decoupling Dynamics and Reward for Transfer LearningPreprint
Zhang, A., Satija, H., & Pineau, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.10689 · Cited by 54 · see citing works
Learning to Adapt: Meta-Learning for Model-Based ControlPreprint
Clavera, I., Nagabandi, A., Fearing, R. S., Abbeel, P., Levine, S., & Finn, C. (2018) · arXiv (Cornell University) · Cited by 45 · see citing works
Robust and Efficient Transfer Learning with Hidden Parameter Markov Decision Processes
Killian, T. W., Konidaris, G., & Doshi‐Velez, F. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11065 · Cited by 8 · see citing works
Synthesized Policies for Transfer and Adaptation across Tasks and EnvironmentsPreprint
Hu, H., Chen, L., Gong, B., & Sha, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03276 · Cited by 8 · see citing works
Efficient transfer learning and online adaptation with latent variable models for continuous controlPreprint
Perez, C. F., Such, F. P., & Karaletsos, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.03399 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
2 of the 33 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224 · see citing works
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193 · see citing works
“This formalism also builds on those presented by Doshi-Velez and Konidaris (2016), Perez et al. (2020), but we extend them to consider different distributions over context parameters; we include both observed and unobserved contexts settings; we include cases where the context is controllable; and…”
BackgroundMethodInfluentialContrastive Behavioral Similarity Embeddings for Generalization in Reinforcement LearningPreprint
Agarwal, R., Machado, M. C., Castro, P. S., & Bellemare, M. G. (2021) · International Conference on Learning Representations · doi:10.48550/arxiv.2101.05265 · Cited by 204 · see citing works
Structure in Deep Reinforcement Learning: A Survey and Open Problems
Mohan, A., Zhang, A., & Lindauer, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15703 · Cited by 26 · see citing works
Toward AI assistants that let designers design
Peuter, S. D., Oulasvirta, A., & Samuel Kaski, A. (2023) · AI Magazine · doi:10.1002/aaai.12077 · Cited by 24 · see citing works
Learning Robust State Abstractions for Hidden-Parameter Block MDPsPreprint
Zhang, A., Sodhani, S., Khetarpal, K., & Pineau, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07206 · Cited by 17 · see citing works
MELD: Meta-Reinforcement Learning from Images via Latent State ModelsPreprint
Zhao, T. Z., Nagabandi, A., Rakelly, K., Finn, C., & Levine, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13957 · Cited by 11 · see citing works
Tütüncüoğlu, F., Ben-Ameur, A., Dán, G., Araldo, A., & Chahed, T. (2024) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs60910.2024.00068 · Cited by 10 · see citing works
Data-Efficient Task Generalization via Probabilistic Model-Based Meta Reinforcement Learning
Bhardwaj, A., Rothfuss, J., Sukhija, B., As, Y., Hutter, M., Coros, S., & Krause, A. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3371260 · Cited by 9 · see citing works
Shared dynamics learning for large-scale traveling salesman problem
Xu, Y., Fang, M., Chen, L., Du, Y., Xu, G., & Zhang, C. (2023) · Advanced Engineering Informatics · doi:10.1016/j.aei.2023.102005 · Cited by 5 · see citing works
Meta-Model-Based Meta-Policy OptimizationPreprint
Hiraoka, T., Imagawa, T., Tangkaratt, V., Osa, T., Onishi, T., & Tsuruoka, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.02608 · Cited by 5 · see citing works
Generalizable Causal Reinforcement Learning for Out-of-Distribution Environments
Huang, S., Hu, J., Chen, H., Cui, P., Piao, H., Sun, L., & Yang, B. (2025) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2025.3556029 · Cited by 4 · see citing works
Multi-Task Reinforcement Learning as a Hidden-Parameter Block MDPPreprint
Zhang, A., Sodhani, S., Khetarpal, K., & Pineau, J. (2020) · arXiv (Cornell University) · Cited by 4 · see citing works
Block Contextual MDPs for Continual Learning
Sodhani, S., Meier, F., Pineau, J., & Zhang, A. (2021) · arXiv · doi:10.48550/arxiv.2110.06972 · Cited by 3 · see citing works
Zero-Shot Assistance in Sequential Decision Problems
Peuter, S. D., & Samuel Kaski, A. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i10.26365 · Cited by 2 · see citing works
Amortized Inference for Efficient Grasp Model Adaptation
Noseworthy, M., Shaw, S., Kessens, C. C., & Roy, N. (2024) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra57147.2024.10610789 · Cited by 3 · see citing works
Multi-policy Grounding and Ensemble Policy Learning for Transfer Learning with Dynamics Mismatch
Lee, H.-R., Sreenivasan, R. A., Jeong, Y., Jang, J., Shim, D., & Lee, C.-G. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/440 · Cited by 1 · see citing works
On the Benefits of Leveraging Structural Information in Planning Over the Learned Model
Shen, J., Kuwaranancharoen, K., Ayoub, R. Z., Mercati, P., & Sundaram, S. (2023) · Proceedings of the ... American Control Conference/Proceedings of the American Control Conference · doi:10.23919/acc55779.2023.10155973 · Cited by 1 · see citing works
GalilAI: Out-of-Task Distribution Detection using Causal Active Experimentation for Safe Transfer RLPreprint
Sontakke, S., Iota, S., Hu, Z., Mehrjou, A., Itti, L., & Schölkopf, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15489 · Cited by 1 · see citing works
Time-Variant Variational Transfer for Value FunctionsPreprint
Canonaco, G., Soprani, A., Roveri, M., & Restelli, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.12864
Experience Filter: Using Past Experiences on Unseen Tasks or Environments
Yildiz, A., Yel, E., Corso, A., Wray, K. H., Witwicki, S., & Kochenderfer, M. J. (2023) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv55152.2023.10186722
Situationally-aware dynamics learning
Murillo-González, A., & Liu, L. (2026) · The International Journal of Robotics Research · doi:10.1177/02783649261431863 · Cited by 1 · see citing works
Dynamic epidemic control with continual learning via mass testing
Yu, H., Du, M., Zhang, Y., & Kong, N. (2026) · IISE Transactions · doi:10.1080/24725854.2026.2682881
- Authors
- Perez, C., Petroski Such, F., & Karaletsos, T.
- Affiliations
- Uber AI (United States) (US)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2020-04-03
- Volume
- 34
- Issue
- 04
- Pages
- 5403-5411
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2996795455
- Cited by
- 25
- References
- 33
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 5 · 2023: 7 · 2022: 2 · 2021: 3 · 2020: 5
- FWCI
- 1.62
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- www.aaai.org/ (unspecified)
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- model-based reinforcement learning, latent variables, task transfer, sample efficiency, hierarchical models, MuJoCo
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
