| AI is making its way into our workflows through text, code, images, and increasingly audio and video. But in the physical world, in factories or self-driving cars, even advanced reasoning models and personal agents aren’t enough. Their weak spot is reliably predicting what happens in the physical world. | World models try to capture how the world changes, including what might happen when an agent takes an action. That could help a robot predict whether pushing an object will move it or tip it over. | But there’s no universally accepted definition. The term covers different approaches, from simulators and video generators to representation models like JEPA. We just came back from a world model workshop, and every participant had their own definition. Several studies have tried to organize the field (we’ll link them at the end), but the categories widely overlap. | So how are these models built? We’ll move from Dreamer 4 and JEPA to Genie 3, NVIDIA’s Cosmos-Predict2.5, and World Labs’ Atlas, then look at what these ideas become in practice: Waymo’s world model for autonomous-driving simulation, agents that practice inside Minecraft, and object-centric models for robotics. Our goal is to explain the architectures in a way that’s easy to grasp and use, so you can judge what each can do, where it falls short, and which might suit what you’re building. |
|
| | | In today’s episode: | The core patterns of World Models: everything comes from dynamics (render, simulate, plan) The core properties of world models World model architectures: how AI represents and predicts the world Observation-Generating World Models: what will happen in the next frame? Latent Dynamics Models: predicting without rendering every future Giving the world structure: Spatial and Object-Centric Models Which World Model architecture should you choose?
|
| |
|
| | | | The core patterns of world models: everything comes from dynamics | Before we jump into the classification of world models, let’s start with the overall patterns and functions. | World models interact with the world and its rules. The world never stays in one state for long. It is a sequence of states that change over time. For example, if we look from the side of mechanics, objects are at rest or move, and we can imagine this process as snapshots going one after another, every second, for instance. At each step, there will be a new state of an object. Then, from the side of science, we have chemicals that react and transform their state, cells that also change, and living systems responding to their surroundings. This is the varying physical world. But in a digital world, we also witness a number of states following each other – like how a system changes before and after running the code or with each line of it. A world model tries to learn the rules behind these transitions: not only what the world looks like now, but also what can happen next and why. | Depending on the environment, these rules may come from physics, chemistry, biology, human behavior, coding or all of them at once. But the general workflow stays the same: | The world has a state: everything that is happening at a particular moment. The agent receives observations: a partial view of that state like images or sensor readings. The world model uses these observations to estimate the current state and predict how it may change. Based on those predictions, the agent takes an action. The action changes the world according to its physical dynamics, producing a new state and new observations.
| In other words, a world model tries to capture and predict world dynamics. It compresses large amounts of visual and physical experience (often buried in pixels) into a smaller, useful understanding of objects, movement, cause and effect, and physical constraints. And another hard task, it needs to keep the important physical patterns while ignoring details like lighting, texture, and background noise. | As Dr. Fei-Fei Li described in her post on Substack, world models have three main functions: | Render: generate what the world looks like. Video models fall into this group. But their output may look realistic without being structurally or physically correct. Simulate: represent how the world is actually built and how it behaves. This includes its geometry, physics, and movement. These models can be used to train robots, test autonomous vehicles, or design buildings. Plan: decide what action to take based on what the model sees and the goal it has been given.
| Simulation connects the other two. If a model understands an object’s shape, physical properties, and response to force, it should be able to show that object from different angles, predict what happens when it is moved, and plan how a robot could interact with it. | To make these tasks easier to perform, the following properties play a huge role for world models: | Be an omnimodal: it is when models can combine different types of information, such as images, sound, text, and sensor readings. Be asynchronous: models need to be able to process signals that arrive at different speeds and times. Be local: a world model works from the agent’s limited view.
| Today, the main functions are usually handled by different world models. Renderers are already widely used, while simulators and planners are much less developed. The long-term goal is to combine all three into one model that can generate a world, simulate it accurately, and act inside it at the same time. | To see how we might get there, let’s unpack how different types of world models work. | World Model architectures: how AI represents and predicts the world | In general, a world model needs a way to represent the current situation and predict what can happen next. For an agent, that prediction also needs to account for its actions. To be honest, there is no general taxonomy to classify world models because many architectural and workflow aspects often meet in one model. | Though, we propose to organize them based on the central architectural question – what the model carries forward: a generated observation, a compact latent state, or an explicit structure of objects and space? → | What does this all mean? | A generated observation – the model carries the world forward by generating what the next observation should look like, such as the next image, video frame, or sequence of tokens. It represents the future by recreating what an observer would see. | A compact latent state – instead of generating the full observation, the model compresses the important information about the current world into an internal representation and predicts how that representation will change. The state is meaningful to the model but is not necessarily directly interpretable by humans. | An explicit structure of objects and space – the model represents the world through identifiable entities and their relationships: objects, their properties, positions, geometry, and interactions. It carries this structured representation forward as the world changes. | Family | Architecture | Core mechanism | Examples |
|---|
Observation-generating | Visual tokens | Predicts future visual tokens autoregressively | IRIS; Dreamer 4 | Observation-generating | Diffusion / flow | Generates future observations through denoising or flow | DIAMOND; Genie 3; Cosmos-Predict2.5 | Latent dynamics | Reconstructive state-space | Predicts compact states and reconstructs observations | PlaNet; Dreamer V1–V3; DayDreamer; HaM-World; DMWM | Latent dynamics | Predictive representations | Predicts future features without reconstructing pixels | JEPA; DINO-WM | Latent dynamics | Task-oriented | Learns transitions useful for reward and value prediction | MuZero; EfficientZero V2; Newt | Structured worlds | Spatial / geometry-aware | Represents depth, occupancy, position, and camera geometry | OccWorld; Atlas | Structured worlds | Object-centric | Tracks separate entities and their interactions | C-SWM; SlotFormer; SlotDiT; TextOCVP; STICA; FIOC-WM |
| But the lines between these three groups aren’t clear-cut: Dreamer imagines long sequences of actions, Waymo generates rare driving scenarios, and Atlas turns sparse observations into explorable space. Let’s look inside the architectures that make this possible → |
| |
|
| Don’t settle for shallow articles. Learn the basics and go deeper with us. Find inspiration for what to build and the knowledge to put it into practice. | | Join Premium members from top companies like Microsoft, NVIDIA, Google, HF, OpenAI, a16z, plus AI labs such as Ai2, MIT, Berkeley, .gov, and thousands of others to really understand what’s going on in AI. |
|
| | | | |
|
