CoChat
Sign Up
PreprintCited by 11Reinforcement Learning in RoboticsRobot Manipulation and LearningDomain Adaptation and Few-Shot Learning

Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment

Open access
Ball, Philip J., Lu, Cong, Parker-Holder, Jack, Roberts, Stephen·2021·arXiv
DOI 10.48550/arxiv.2104.05632arXiv 2104.05632
Open this paper in CoChat — free accountOpen PDF

Abstract

Reinforcement learning from large-scale offline datasets provides us with the ability to learn policies without potentially unsafe or impractical exploration. Significant progress has been made in the past few years in dealing with the challenge of correcting for differing behavior between the data collection and learned policies. However, little attention has been paid to potentially changing dynamics when transferring a policy to the online setting, where performance can be up to 90% reduced for existing methods. In this paper we address this problem with Augmented World Models (AugWM). We augment a learned dynamics model with simple transformations that seek to capture potential changes in physical properties of the robot, leading to more robust policies. We not only train our policy in this new setting, but also provide it with the sampled augmentation as a context, allowing it to adapt to changes in the environment. At test time we learn the context in a self-supervised fashion by approximating the augmentation which corresponds to the new environment. We rigorously evaluate our approach on over 100 different changed dynamics settings, and show that this simple approach can significantly improve the zero-shot generalization of a recent state-of-the-art baseline, often achieving successful policies where the baseline fails.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2104.05632

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Explicit context conditioning approaches scale poorly and break down in unstructured environments where contexts are latent or difficult to measure.

    “While effective in controlled settings, such approaches scale poorly and break down in unstructured environments where contexts are latent or difficult to measure.”

    From Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalization (2025) · cites this paper as Ball et al., 2021

    The Discussion (p. 8) says context-based training generally takes longer to converge and learned-context performance was reduced in one more nonlinear setting. It does not establish that explicit context conditioning generally scales poorly or breaks down in unstructured environments where contexts are latent or difficult to measure; Section 4.2 (p. 5) instead describes a setting where the context is explicitly known and learned on the fly. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Existing contextual RL methods often require explicit context variables, limiting their use when contexts are latent or hard to measure.

    “Contextual Markov Decision Processes (cMDP) model this challenge, but existing methods often require explicit context variables (e.g., friction, gravity), limiting their use when contexts are latent or hard to measure.”

    From Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalization (2025) · cites this paper as Ball et al., 2021

    The Introduction (p. 2) says the authors’ method works without access to environment parameters, supporting the relevance of unmeasured contexts. But the paper does not say existing contextual RL methods often require explicit context variables; Section 4.2 (p. 5) describes prior methods that learn a context using a latent state and identifies their need for several adaptation episodes as the limitation. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. Ajay, A., Kumar, A., Agrawal, P., Levine, S., and Nachum, O. (2021). OPAL: Offline primitive discovery for accelerating offline reinforcement learning. In International Conference on Learning Representations .
  2. Welinder, P., Weng, L., and Zaremba, W. (2020). Learning dexterous in-hand manipulation. The International Journal of Robotics Research , 39(1):3-20.
  3. Argenson, A. and Dulac-Arnold, G. (2021). Model-based offline planning. In International Conference on Learning Representations .
  4. Ball, P., Parker-Holder, J., Pacchiano, A., Choromanski, K., and Roberts, S. (2020). Ready policy one: World building through active learning. In Proceedings of the 37th International Conference on Machine Learning, ICML .
  5. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym. arXiv preprint arXiv:1606.01540 . · arXiv
  6. Chua, K., Calandra, R., McAllister, R., and Levine, S. (2018). Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems 31 , pages 47544765.
  7. Clavera, I., Rothfuss, J., Schulman, J., Fujita, Y ., Asfour, T., and Abbeel, P. (2018). Model-based reinforcement learning via meta-policy optimization. In 2nd Annual Conference on Robot Learning, CoRL 2018, Z¨ urich, Switzerland, 29-31 October 2018, Proceedings , volume 87 of Proceedings of Machine Learning Research , pages 617-629. PMLR.
  8. Doshi-Velez, F. and Konidaris, G. (2016). Hidden parameter markov decision processes: A semiparametric regression approach for discovering latent task parametrizations. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence , IJCAI'16, page 1432-1440. AAAI Press.
  9. Dulac-Arnold, G., Mankowitz, D. J., and Hester, T. (2019). Challenges of real-world reinforcement learning. CoRR , abs/1904.12901. · arXiv
  10. Fujimoto, S., Meger, D., and Precup, D. (2019). Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning , pages 20522062.
  11. Gu, S., Lillicrap, T., Sutskever, I., and Levine, S. (2016). Continuous deep q-learning with model-based acceleration. In Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 2829-2838. PMLR.
  12. Ha, D. and Schmidhuber, J. (2018). Recurrent world models facilitate policy evolution. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems , NeurIPS'18, pages 2455-2467.
  13. Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. (2020). Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations .
  14. Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. (2019). Learning latent dynamics for planning from pixels. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference on Machine Learning , pages 2555-2565.
  15. Hansen, N., Jangir, R., Sun, Y., Aleny` a, G., Abbeel, P., Efros, A. A., Pinto, L., and Wang, X. (2021). Selfsupervised policy adaptation during deployment. In International Conference on Learning Representations .
  16. Henderson, P., Chang, W.-D., Shkurti, F., Hansen, J., Meger, D., and Dudek, G. (2017). Benchmark environments for multitask learning in continuous domains. ICML Lifelong Learning: A Reinforcement Learning Approach Workshop .
  17. James, S., Davison, A. J., and Johns, E. (2017). Transferring end-to-end visuomotor control from simulation to real world for a multi-stage task. In 1st Conference on Robot Learning .
  18. Janner, M., Fu, J., Zhang, M., and Levine, S. (2019). When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems .
  19. Tucker, G., and Michalewski, H. (2020). Model based reinforcement learning for Atari. In International Conference on Learning Representations .
  20. Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. (2020). Morel : Model-based offline reinforcement learning. In Advances in Neural Information Processing Systems .
  21. Killian, T., Konidaris, G., and Doshi-Velez, F. (2017). Robust and efficient transfer learning with hidden parameter markov decision processes. Proceedings of the AAAI Conference on Artificial Intelligence , 31(1).
  22. Kostrikov, I., Yarats, D., and Fergus, R. (2021). Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations .
  23. Kumar, A., Zhou, A., Tucker, G., and Levine, S. (2020). Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems .
  24. Kurutach, T., Clavera, I., Duan, Y., Tamar, A., and Abbeel, P. (2018). Model-ensemble trust-region policy optimization. In International Conference on Learning Representations .
  25. Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems , volume 30.
  26. Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., and Srinivas, A. (2020a). Reinforcement learming with augmented data. In Advances in Neural Information Processing Systems 33 .
  27. Laskin, M., Srinivas, A., and Abbeel, P. (2020b). CURL: Contrastive unsupervised representations for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning .
  28. Levine, S., Kumar, A., Tucker, G., and Fu, J. (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems.
  29. Modi, A., Jiang, N., Singh, S., and Tewari, A. (2018). Markov decision processes with continuous side information. In Proceedings of Algorithmic Learning Theory , volume 83 of Proceedings of Machine Learning Research , pages 597-618. PMLR.
  30. Nagabandi, A., Finn, C., and Levine, S. (2019). Deep online learning via meta-learning: Continual adaptation for model-based RL. In International Conference on Learning Representations .
  31. Nix, D. A. and Weigend, A. S. (1994). Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94) , volume 1, pages 55-60 vol.1.
  32. OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., Schneider, J., Tezak, N., Tworek, J., Welinder, P., Weng, L., Yuan, Q., Zaremba, W., and Zhang, L. (2019). Solving rubik's cube with a robot hand. CoRR , abs/1910.07113. · arXiv
  33. Peng, M., Zhu, B., and Jiao, J. (2021). Linear representation meta-reinforcement learning for instant adaptation. CoRR .
  34. Peng, X. B., Andrychowicz, M., Zaremba, W., and Abbeel, P. (2018). Sim-to-real transfer of robotic control with dynamics randomization. In IEEE International Conference on Robotics and Automation, ICRA .
  35. Pitis, S., Creager, E., and Garg, A. (2020). Counterfactual data augmentation using locally factored dynamics. In Advances in Neural Information Processing Systems .
  36. Raileanu, R., Goldstein, M., Yarats, D., Kostrikov, I., and Fergus, R. (2020). Automatic data augmentation for generalization in deep reinforcement learning. CoRR , abs/2006.12862. · arXiv
  37. Rajendran, J., Irpan, A., and Jang, E. (2020). Meta-learning requires meta-augmentation. In Advances in Neural Information Processing Systems .
  38. Rakelly, K., Zhou, A., Finn, C., Levine, S., and Quillen, D. (2019). Efficient off-policy meta-reinforcement learning via probabilistic context variables. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019 , volume 97, pages 5331-5340. PMLR.
  39. Rudner, T. G. J., Lu, C., Osborne, M. A., Gal, Y., and Teh, Y. W. (2021). On pathologies in kl-regularized reinforcement learning from expert demonstrations. ICLR RobustML Workshop .
  40. Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap, T. P., and Silver, D. (2019). Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. CoRR , abs/1911.08265. · arXiv
  41. Seo, Y., Lee, K., Clavera, I., Kurutach, T., Shin, J., and Abbeel, P. (2020). Trajectory-wise multiple choice learning for dynamics generalization in reinforcement learning. In Advances in Neural Information Processing Systems .
  42. Song, X., Jiang, Y., Tu, S., Du, Y ., and Neyshabur, B. (2020). Observational overfitting in reinforcement learning. In International Conference on Learning Representations .
  43. Volpi, R., Namkoong, H., Sener, O., Duchi, J. C., Murino, V., and Savarese, S. (2018). Generalizing to unseen domains via adversarial data augmentation. In Advances in Neural Information Processing Systems , volume 31, pages 53345344. Curran Associates, Inc.
  44. Wellmer, Z. and Kwok, J. T. (2021). Dropout's dream land: Generalization from learned simulators to reality. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases . Springer.
  45. Wu, Y., Tucker, G., and Nachum, O. (2019). Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 . · arXiv
  46. Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C., and Ma, T. (2020). Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems .
  47. Zhang, A., Sodhani, S., Khetarpal, K., and Pineau, J. (2021). Learning robust state abstractions for hidden-parameter block mdps. In International Conference on Learning Representations .
  48. Zhou, W., Pinto, L., and Gupta, A. (2019). Environment probing interaction policies. In International Conference on Learning Representations .
  49. Zintgraf, L., Shiarlis, K., Igl, M., Schulze, S., Gal, Y ., Hofmann, K., and Whiteson, S. (2020). Varibad: A very good method for bayes-adaptive deep rl via meta-learning. In International Conference on Learning Representations .
  50. Zintgraf, L. M., Feng, L., Lu, C., Igl, M., Hartikainen, K., Hofmann, K., and Whiteson, S. (2021). Exploration in approximate hyper-state space for meta reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning , volume 139, pages 12991-13001. PMLR.

Cited by

Sorted by citations

Authors
Ball, P. J., Lu, C., Parker-Holder, J., & Roberts, S.
Affiliations
University of Oxford (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Pages
619-629
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2104.05632
arXiv
2104.05632
OpenAlex
W3154412344
Cited by
11
References
6
Citations by year
2025: 2 · 2024: 4 · 2023: 2 · 2022: 1 · 2021: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
  • International Conference on Machine Learning
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
offline reinforcement learning, learned dynamics models, physical property changes, online deployment, domain adaptation, robot control

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review