JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Dreaming in Latent Space: World Models

When the world is high-dimensional pixels, compress it into a latent code and learn — even dream — entirely inside it.

Why pixels break naive models

Predicting the next raw image is brutally hard and mostly pointless — most pixels are background that has nothing to do with the decision. Trying to forecast every pixel wastes capacity and amplifies compounding error. The fix is to stop modelling pixels directly and instead learn a compact summary of what matters.

Latent dynamics: model the gist, not the picture

A latent dynamics model first encodes each observation into a small latent state — a learned compression that keeps the decision-relevant content and throws away noise — and then predicts how that latent state evolves over time. Planning and imagination now happen in this tidy latent space, which is faster, smoother, and far more forgiving than pixel space. The encoder is typically trained with the same machinery as a variational autoencoder.

Like a VAE, the latent dynamics model encodes each observation into a small latent code and can decode it back — keeping the decision-relevant content while throwing away noise.

An encoder maps an input to a compact latent vector; a decoder reconstructs the input from that latent.

World models: a learned simulator

Stitch the pieces together — an encoder, a latent transition model, a reward predictor, and often a decoder that can reconstruct images — and you get a world model: a complete learned simulator of the environment that runs entirely in the agent's head. Once you own such a simulator, you can generate as much imagined experience as compute allows, the ultimate expression of sample efficiency.

\hat{s}_{t+1}\sim p_\theta(\hat{s}_{t+1}\mid \hat{s}_t, a_t),\qquad \hat{r}_t\sim p_\theta(\hat{r}_t\mid \hat{s}_t)

The heart of a world model: a learned simulator that predicts the next latent state and the reward, so the agent can plan without touching the real environment.

Dreamer: learning purely from imagination

Dreamer takes this to its logical end. It learns a latent world model from a replay of past experience, then trains its actor and critic entirely on imagined latent rollouts — it literally dreams thousands of trajectories without touching the environment. Crucially, because the latent state is low-dimensional, these dreams are cheap, and gradients can flow back through the imagined trajectory to improve the policy directly.

Dreamer imagines long rollouts by stepping the latent state forward like a Markov chain — training its actor and critic entirely inside these dreamed trajectories.

Latent states linked by transition arrows, rolled forward step by step into an imagined trajectory.

  1. Encode real experience from a replay buffer into latent states and train the world model to predict latent transitions and reward.
  2. Imagine long latent rollouts from those states using only the model.
  3. Learn the actor and critic on the imagined rollouts, backpropagating through the dynamics.
  4. Act with the improved actor in the real world to gather a little new data, then repeat.

Strengths and honest limits

Latent world models deliver striking data efficiency on visual control and have scaled to diverse tasks from a single agent. But they inherit every hazard from earlier guides: if the encoder drops something the reward secretly depends on, the dream becomes confidently wrong. Reconstruction-based world models can also waste effort modelling visually busy but decision-irrelevant detail — a tension the final guide resolves by asking the model to be accurate only about what matters for value.