A world model is a machine learning system trained to predict how a physical or simulated environment evolves, given its current state and an action taken within it. Instead of just classifying or generating a static output, a world model learns dynamics: physics, object permanence, cause and effect, and how a scene changes frame to frame.
The term dates back to Jürgen Schmidhuber's early 1990s work on predictive world models for reinforcement learning agents, and was popularized more recently in the 2018 "World Models" paper by Ha and Schmidhuber.
The 2025–2026 wave, driven by easy availability of video data, better architectures, and massive compute, has turned the idea from a research curiosity into deployed infrastructure for robotics and autonomous systems.
A world model differs from a general video generation model in one key way: it's interactive and state-consistent. A world model like Google's DeepMind Genie or NVIDIA Cosmos generates an environment that responds to actions, an agent can move through it, and the model has to keep track of what's behind the camera, what an object was doing a moment ago, and how a physical interaction should resolve.
How Do World Models Work?
World models are trained on large volumes of video, sensor, and vision-language action model data to learn a compressed internal representation of an environment, then use that representation to predict future states.
Current approaches include:
- Autoregressive latent diffusion, where the model predicts the next frame or state token by token, conditioned on previous frames and the action taken
- JEPA (Joint Embedding Predictive Architecture), championed by Yann LeCun, which predicts future representations in an abstract embedding space rather than pixel space, aiming for more efficient, generalizable world understanding
- VAE + recurrent architectures, the original Ha and Schmidhuber approach, still used in lightweight or research settings
What Makes World Model Training Data Different?
World model training data has to capture dynamics, not just content. That means every example needs more than a single labeled frame:
- Action-conditioned sequences: paired observations and the actions taken between them, not just a single labeled image or clip
- Physically consistent, multi-sensor data: video alone tells you what a scene looks like; pairing it with LiDAR, depth, or force data grounds a model in what a scene actually is, geometrically and physically
- Long-horizon consistency: sequences long enough for a model to learn that an object leaving the frame still exists, and that an action taken now has consequences several seconds later
- Diverse and edge-case coverage: a world model that's only seen normal driving or normal grasps fails exactly where it matters, the near-miss, the unusual object pose, the rare failure mode, so training sets need deliberate edge-case mining, not just volume
Poorly annotated or inconsistent action labels teach a world model the wrong dynamics, which is a harder failure to catch than a mislabeled bounding box, because it doesn't show up until the model is asked to predict several steps ahead.
Encord for World Model Training Data
Encord supports the annotation and curation workflows that world model training depends on: synchronised multi-camera and sensor fusion video, frame-accurate action and event labeling, and trajectory annotation across long, continuous sequences. Curation tools built on embedding search help teams surface the rare scenarios, unusual object interactions, and edge-of-distribution physical events, underrepresented in raw footage but critical for a model to generalise beyond its training distribution.
Quality workflows support consensus review and relabeling loops, so action and state labels stay consistent across large-scale, multimodal datasets.
→ Explore Encord for Physical AI