Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index; 18 of the 56 sources its claims cite were checked for support.

Mastering diverse control tasks through world models

Open access
Hafner, Danijar, Pasukonis, Jurgis, Ba, Jimmy, Lillicrap, Timothy · 2025 · Nature

Abstract

DreamerV3 is a general algorithm that outperforms specialized methods across over 150 diverse tasks, with a single configuration. Dreamer learns a model of the environment and improves its behavior by imagining future scenarios. Robustness techniques based on normalization, balancing, and transformations enable stable learning across domains.

Overview

CoChat Summary

TL;DR

DreamerV3 is a general model-based reinforcement-learning algorithm that learns a world model and trains actor and critic in imagination, using robustness techniques to work with fixed hyperparameters across over 150 tasks in 8 domains. It matches or exceeds specialized experts and is, to the authors’ knowledge, the first algorithm to collect diamonds in Minecraft from scratch without human data or curricula.

Key findings

  • Across 8 domains and over 150 tasks with fixed hyperparameters, Dreamer matches or exceeds tuned experts and outperforms PPO in all domains: Atari (57 games, 200 million frames) beats MuZero, Rainbow and IQN; ProcGen (16 games, 50 million frames) beats PPG and Rainbow; DMLab (30 tasks, 100 million frames) exceeds IMPALA and R2D2+ at 1 billion steps, a data-efficiency gain of over 1,000%; Atari100k (26 games, 400,000 frames) beats IRIS, TWM, SimPLe and SPR; Proprio Control Suite (20 tasks, 1 million steps) matches DMPO and TD-MPC2; Visual Control Suite (20 tasks) sets state of the art over DrQ-v2 and TD-MPC2; BSuite (23 environments, 468 configurations) sets state of the art over Boot DQN.
  • In Minecraft, all Dreamer agents discover diamonds within 100 million environment steps, completing all 12 milestones from sparse rewards without human data or curricula; Dreamer uses 1 GPU for 9 days, whereas VPT used 720 GPUs for 9 days.
  • Ablations on 14 tasks show all robustness techniques contribute, most notably KL balancing and free bits, followed by return normalization and symexp two-hot regression; performance rests predominantly on the unsupervised reconstruction loss of the world model.
  • Scaling from 12 million to 400 million parameters increases task performance monotonically and reduces required environment interaction; higher replay ratios also predictably improve performance.

Methods

  • DreamerV3 consists of three neural networks trained concurrently from replayed experience: a recurrent state-space world model that encodes sensory inputs into discrete representations, predicts future representations and rewards, and reconstructs inputs; a critic that predicts return distributions; and an actor that selects actions to maximize return.
  • Robustness techniques include symlog observation and target transforms, symexp two-hot loss for reward and critic, KL balancing with free bits, 1% unimix for categorical distributions, percentile return normalization, block GRU, RMSNorm, SiLU activations, adaptive gradient clipping, LaProp optimizer, and a uniform replay buffer with online queue; hyperparameters are fixed across all benchmarks.
  • Evaluation spans 8 domains with over 150 tasks, comparing against PPO and specialized experts; seeds are 5 per benchmark except 10 for BSuite and Minecraft; all agents train on a single Nvidia A100 GPU. Minecraft uses MineRL v0.4.4, a 64×64×3 first-person image, inventory and scalar observations, 25 categorical actions, sparse rewards for 12 milestones, episodes up to 36,000 steps (30 minutes), and accelerated block breaking.

Limitations

  • Stated: DMLab baselines were not designed for data efficiency and were compared at 1 billion steps while Dreamer used 100 million; EfficientZero resets levels early, making comparison difficult. Minecraft uses accelerated block breaking and abstract crafting actions, and the authors did not investigate whether the heart reward is helpful.
  • Stated: uniform replay was chosen for ease of implementation even though prioritized replay improved Dreamer; no annealing, prioritized replay, weight decay or dropout were used.
  • Evident: Minecraft environment modifications and the categorical action space mean results are not directly comparable to unmodified Minecraft or to VPT/Voyager setups; evaluation budgets were limited by computational cost, such as DMLab at 100 million rather than 10 billion steps and ProcGen at 50 million steps.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41586-025-08744-2

Cites

50 of 50 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Training process of Dreamer. The world model encodes sensory inputs xt using the encoder (enc) into discrete representations zt that are predicted by a sequence model with recurrent state ht given actions at. The inputs are reconstructed as x^t using the decoder (dec) to shape the representations. The actor and critic predict actions at and values vt and learn from trajectories of abstract representations z^t and rewards rt predicted by the world model.
Fig. 1 Training process of Dreamer. The world model encodes sensory inputs xt using the encoder (enc) into discrete representations zt that are predicted by a sequence model with recurrent state ht given actions at. The inputs are reconstructed as x^t using the decoder (dec) to shape the representations. The actor and critic predict actions at and values vt and learn from trajectories of abstract representations z^t and rewards rt predicted by the world model.CC BY · PubMed Central
Fig. 2 Diverse visual domains used in the experiments. Dreamer succeeds across these domains, ranging from robot locomotion and manipulation tasks over Atari games, procedurally generated ProcGen levels, and DMLab tasks, which require spatial and temporal reasoning, to the complex and infinite world of Minecraft. We also evaluate Dreamer on non-visual domains.
Fig. 2 Diverse visual domains used in the experiments. Dreamer succeeds across these domains, ranging from robot locomotion and manipulation tasks over Atari games, procedurally generated ProcGen levels, and DMLab tasks, which require spatial and temporal reasoning, to the complex and infinite world of Minecraft. We also evaluate Dreamer on non-visual domains.CC BY · PubMed Central
Fig. 3 Video predictions of the world model. A procedural maze and a quadrupedal robot are shown. Given 5 context images and the full action sequence of an unseen video, Dreamer predicts 45 frames into the future without access to intermediate images. From pixel observations, the world model learns an understanding of the underlying structure of each environment.
Fig. 3 Video predictions of the world model. A procedural maze and a quadrupedal robot are shown. Given 5 context images and the full action sequence of an unseen video, Dreamer predicts 45 frames into the future without access to intermediate images. From pixel observations, the world model learns an understanding of the underlying structure of each environment.CC BY · PubMed Central
Fig. 4 Benchmark scores. Using fixed hyperparameters across all domains, Dreamer outperforms tuned expert algorithms across a wide range of benchmarks and data budgets. Dreamer also substantially outperforms a high-quality implementation of the widely applicable PPO algorithm. IMPALA and R2D2+ use ten times more data on DMLab.
Fig. 4 Benchmark scores. Using fixed hyperparameters across all domains, Dreamer outperforms tuned expert algorithms across a wide range of benchmarks and data budgets. Dreamer also substantially outperforms a high-quality implementation of the widely applicable PPO algorithm. IMPALA and R2D2+ use ten times more data on DMLab.CC BY · PubMed Central
Fig. 5 Performance on the Minecraft Diamond challenge. a, Applied out of the box, Dreamer is, to our knowledge, the first algorithm to accomplish all 12 milestones leading up to the diamond, from sparse rewards without human data or curricula. b, Fraction of trained agents that discover each of the three latest items in the diamond task, and the fraction of episodes during which they obtain the item. Although previous algorithms progress up to the iron pickaxe, Dreamer is the only compared algorithm that discovers diamonds, and does so in every training run. Shaded areas indicate one standard deviation.
Fig. 5 Performance on the Minecraft Diamond challenge. a, Applied out of the box, Dreamer is, to our knowledge, the first algorithm to accomplish all 12 milestones leading up to the diamond, from sparse rewards without human data or curricula. b, Fraction of trained agents that discover each of the three latest items in the diamond task, and the fraction of episodes during which they obtain the item. Although previous algorithms progress up to the iron pickaxe, Dreamer is the only compared algorithm that discovers diamonds, and does so in every training run. Shaded areas indicate one standard deviation.CC BY · PubMed Central
Fig. 6 Ablations and robust scaling of Dreamer. a, All individual robustness techniques contribute to the performance of Dreamer on average, although each individual technique may affect only some tasks. Training curves of individual tasks are included in Supplementary Information. advnorm, advantage normalization. b, The performance of Dreamer predominantly rests on the unsupervised reconstruction loss of its world model, unlike most previous algorithms that rely predominantly on reward and value prediction gradients7,9,10. c, The performance of Dreamer increases monotonically with larger model sizes, ranging from 12 million to 400 million parameters. Notably, larger models not only increase task performance but also require less environment interaction. d, Higher replay ratios predictably increase the performance of Dreamer. Together with model size, this allows practitioners to improve task performance and data efficiency by employing more computational resources.
Fig. 6 Ablations and robust scaling of Dreamer. a, All individual robustness techniques contribute to the performance of Dreamer on average, although each individual technique may affect only some tasks. Training curves of individual tasks are included in Supplementary Information. advnorm, advantage normalization. b, The performance of Dreamer predominantly rests on the unsupervised reconstruction loss of its world model, unlike most previous algorithms that rely predominantly on reward and value prediction gradients7,9,10. c, The performance of Dreamer increases monotonically with larger model sizes, ranging from 12 million to 400 million parameters. Notably, larger models not only increase task performance but also require less environment interaction. d, Higher replay ratios predictably increase the performance of Dreamer. Together with model size, this allows practitioners to improve task performance and data efficiency by employing more computational resources.CC BY · PubMed Central
Extended Data Fig. 1 Video predictions of Minecraft. The world model receives the first 5 frames of an unseen video as context input and the predicts 45 steps into the future given the action sequence, without access to intermediate images.
Extended Data Fig. 1 Video predictions of Minecraft. The world model receives the first 5 frames of an unseen video as context input and the predicts 45 steps into the future given the action sequence, without access to intermediate images.CC BY · PubMed Central
Extended Data Fig. 2 Minecraft item success rates. Dreamer obtains items at substantially higher rates than the baselines and continues to improve on hard items throughout training.
Extended Data Fig. 2 Minecraft item success rates. Dreamer obtains items at substantially higher rates than the baselines and continues to improve on hard items throughout training.CC BY · PubMed Central

Verification

Claims the paper makes

13 Valid · 7 Not valid · 27 Not checked · 57 Own results and interpretations

36 more claims

Create a free account to see all 104 claims and to verify the 10 references still open.

See all claims — free account
  • It may state two separate claims as one. 39
  • Probably not a substantial claim. 36
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 39 references

Research works printed but not linked: 20

  1. Huang, S., Dossa, R. F. J., Raffin, A., Kanervisto, A. & Wang, W. The 37 implementation details of proximal policy optimization. The ICLR Blog Track https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/ (2023).
  2. Ha, D. & Schmidhuber, J. Recurrent world models facilitate policy evolution. In Proc. 32nd International Conference on Neural Information Processing Systems 2455–2467 (Association for Computing Machinery, 2018).
  3. Hafner, D., Lillicrap, T., Ba, J. & Norouzi, M. Dream to control: learning behaviors by latent imagination. In Proc. International Conference on Learning Representations (ICLR, 2020).
  4. Kingma, D. P. & Welling, M. Auto-encoding variational Bayes. In Proc. International Conference on Learning Representations (ICLR, 2014).
  5. Child, R. Very deep VAEs generalize autoregressive models and can outperform them on images. In Proc. International Conference on Learning Representations (ICLR, 2021).
  6. Bellemare, M. G., Dabney, W. & Munos, R. A distributional perspective on reinforcement learning. In Proc. 34th International Conference on Machine Learning 449–458 (PMLR, 2017).
  7. Kapturowski, S., Ostrovski, G., Quan, J., Munos, R. & Dabney, W. Recurrent experience replay in distributed reinforcement learning. In Proc. International Conference on Learning Representations (ICLR, 2018).
  8. Cobbe, K. W., Hilton, J., Klimov, O. & Schulman, J. Phasic policy gradient. In Proc. 38th International Conference on Machine Learning 2020–2027 (PMLR, 2021).
  9. Cobbe, K., Hesse, C., Hilton, J. & Schulman, J. Leveraging procedural generation to benchmark reinforcement learning. In Proc. 37th International Conference on Machine Learning 2048–2056 (Association for Computing Machinery, 2020).
  10. Hansen, N., Su, H. & Wang, X. TD-MPC2: scalable, robust world models for continuous control. In Proc. International Conference on Learning Representations (ICLR, 2024).
  11. Yarats, D., Fergus, R., Lazaric, A. & Pinto, L. Mastering visual continuous control: Improved data-augmented reinforcement learning. In Proc. International Conference on Learning Representations (ICLR, 2022).
  12. Hafner, D. Benchmarking the spectrum of agent capabilities. In Proc. International Conference on Learning Representations (ICLR, 2022).
  13. Samsami, M. R., Zholus, A., Rajendran, J. & Chandar, S. Mastering memory tasks with world models. In Proc. International Conference on Learning Representations (ICLR, 2024).
  14. Robine, J., Höftmann, M., Uelwer, T. & Harmeling, S. Transformer-based world models are happy with 100k interactions. In Proc. International Conference on Learning Representations (ICLR, 2023).
  15. Osband, I., Blundell, C., Pritzel, A. & Van Roy, B. Deep exploration via bootstrapped DQN. In Proc. Advances in Neural Information Processing Systems 29 (NeurIPS, 2016).
  16. Brock, A., De, S., Smith, S. L. & Simonyan, K. High-performance large-scale image recognition without normalization. In Proc. 38th International Conference on Machine Learning 1059–1071 (PMLR, 2021).
  17. Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In Proc. International Conference on Learning Representations (ICLR, 2017).
  18. Schmitt, S., Hessel, M. & Simonyan, K. Off-policy actor–critic with shared experience replay. In Proc. 37th International Conference on Machine Learning 8545–8554 (PMLR, 2020).
  19. Schaul, T., Quan, J., Antonoglou, I. & Silver, D. Prioritized experience replay. In Proc. International Conference on Learning Representations (ICLR, 2018).
  20. Sharma, S., Srinivas, A. & Ravindran, B. Learning to repeat: fine grained action repetition for deep reinforcement learning. In Proc. International Conference on Learning Representations (ICLR, 2017).

Cited by

See all 189

Metadata

Authors
Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T.
Affiliations
University of Toronto (CA)
Corresponding author
Hafner, Danijar
Venue
Nature
Published
2025-04-02
Volume
640
Issue
8059
Pages
647-653
ISSN
  • 0028-0836 (ISSN-L)
  • 1476-4687
OpenAlex
W4409147637
Cited by
189
References
39
Citations by year
2026: 145 · 2025: 44
FWCI
75.21
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
dreamers, world models, sparse rewards, domain generalization
MeSH terms
Algorithms, Artificial Intelligence, Humans, Learning, Problem Solving, Reinforcement, Psychology, Task Performance and Analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Mastering diverse control tasks through world models. https://app.cochat.ai/papers/10.1038/s41586-025-08744-2
The paper
Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2025). Mastering diverse control tasks through world models. Nature. https://doi.org/10.1038/s41586-025-08744-2

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.