The Mechanics of Internal Latent Simulation
Predicting the next word in a text sequence works remarkably well for conversational assistants, but it hits a hard wall when an AI system must interact with the physical world. Language models predict tokens; they don't build spatial models of objects falling, colliding, or moving through space. World models alter this equation by constructing internal, compressed representations of visual and temporal dynamics. Instead of reacting blindly to high-dimensional sensory feeds, an agent equipped with a world model builds a structured mental representation of how its environment transitions in response to its actions.
The foundational groundwork for neural world models was established by David Ha and Jürgen Schmidhuber (2018). They proved that generative neural networks can learn low-dimensional latent spaces from raw pixel streams in an unsupervised manner. By combining a Variational Autoencoder (VAE) to compress spatial visual frames into latent vectors with a Recurrent Neural Network (RNN) to model temporal transitions, their system learned to predict future latent states based on current states and candidate actions. Rather than feeding high-dimensional images straight into a policy controller, the agent operated on the model's compact latent vectors. This decoupled visual perception from action selection, enabling lightweight policy networks to master complex tasks like racing and obstacle avoidance.
However, early visual world models suffered from a key computational bottleneck: pixel reconstruction. When a network is trained to reconstruct every pixel of a visual frame, it expends huge capacity predicting high-frequency visual noise—such as leaf flutter, water ripples, or video compression artifacts—that has zero impact on decision-making. To solve this, Meta introduced V-JEPA (Adrien Bardes et al., 2024), shifting the training objective from pixel reconstruction to abstract feature prediction in representation space. By masking parts of video sequences and predicting the latent representations of missing regions rather than raw RGB values, V-JEPA avoids wasting capacity on unpredictable pixel details while capturing underlying motion and temporal continuity. Trained on two million public videos without text supervision or pretrained encoders, a frozen V-JEPA backbone achieved 81.9% accuracy on the Kinetics-400 benchmark, proving that self-supervised feature prediction builds effective visual representations.
Parallel research replaced recurrent architectures with Transformer models to better capture long-horizon dependencies. The IRIS framework (Vincent Micheli et al., 2022) combined discrete autoencoders with autoregressive Transformers. By converting visual patches into discrete latent tokens, IRIS frames environment simulation as a sequence modeling task. The Transformer autoregressively predicts future discrete image tokens and expected rewards, producing a stable simulation engine for complex game dynamics.
Learning in Hallucinations: Control and Generative Environments
Once an AI system can reliably forecast environment transitions inside a compressed latent space, a major opportunity opens up: agents no longer need to execute millions of slow, costly interaction steps in the real world.
Ha and Schmidhuber (2018) showed that an agent can be trained entirely inside its own "hallucinated" dreams generated by its world model (https://arxiv.org/abs/1803.10122). The policy network runs inside synthetic latent rollouts, optimizing its actions against predicted state transitions and reward signals. Once trained within this hallucinated world, the compact policy transfers back to the actual environment, maintaining high task performance without having interacted directly with real physics during policy optimization.
This capability reached a key scaling milestone with DreamerV3 (Danijar Hafner et al., 2023). Earlier reinforcement learning methods required custom, task-specific hyperparameter tuning across different domains. DreamerV3 introduced a unified world model framework stabilized by signal normalization and value transformations. Evaluated across over 150 diverse tasks without per-task parameter tuning, DreamerV3 became the first algorithm to mine diamonds in Minecraft from scratch, operating purely from raw pixel inputs and sparse rewards without human video demonstrations or custom curricula. The agent learns by unrolling prospective futures in latent space, evaluating multi-step action sequences against expected reward trajectories far beyond immediate observations.
The sample efficiency of Transformer-driven latent rollouts is equally compelling. IRIS demonstrated that with just two hours of simulated gameplay experience—equivalent to 100k environment steps—its agent achieved a human-normalized mean score of 1.046 on the Atari 100k benchmark (https://arxiv.org/abs/2209.00588). Outperforming human players on 10 out of 26 games without lookahead search, IRIS showed how sequence modeling in world representations reduces data requirements for deep reinforcement learning.
World models have also expanded into generative interactive media. Google DeepMind’s Genie (Jake Bruce et al., 2024) scales world modeling to an 11-billion parameter foundation model trained unsupervised on unlabelled internet video. Genie combines a spatiotemporal video tokenizer, an autoregressive dynamics model, and a latent action model to infer underlying control inputs without explicit action annotations. Prompted with a single photograph, image, or hand-drawn sketch, Genie generates controllable, interactive 2D environments frame-by-frame. Users and downstream agents can execute discrete actions inside these synthesized virtual worlds, positioning foundation world models as versatile simulation environments.
Temporal Stability, Causal Physics, and Real-World Transfer
Despite these breakthroughs, world models face serious theoretical and physical limitations that separate predictive simulation from true physical reasoning.
The primary limitation is compounding temporal rollout error. When an autoregressive model predicts environment trajectories over extended time horizons, small latent estimation errors accumulate. Over short rollout windows, predicted latent states remain sharp and physically plausible. However, as rollout steps extend into hundreds or thousands of frames, accumulated errors degrade the generated latent space, causing hallucinated objects to disappear, solid boundaries to fail, or visual representations to collapse.
Second, visual world models struggle to separate observational correlations from true causal physical simulation. A model trained on video clips might predict that a glass dropped from a counter will shatter when it hits the ground. But it makes this prediction by recognizing statistical correlations across video data, not by calculating gravitational force, material strength, or kinetic impact dynamics. When presented with counterfactual situations—such as a glass falling onto a thick foam pad—purely visual models often rely on statistical defaults and hallucinate a shatter event anyway, revealing a lack of explicit causal physical grounding.
Finally, the sim-to-real gap remains a persistent obstacle for robotic hardware deployment. While an agent can learn high-performing policies inside DreamerV3 or IRIS latent spaces, physical deployment brings sensory noise, joint latencies, and micro-surface friction variations that compressed latent spaces fail to represent perfectly. If a latent world model miscalculates joint resistance or surface friction by even a small margin, a policy optimized inside that hallucinated environment can fail or oscillate violently when executed on real robotic hardware.
Solving these open challenges will require hybrid model designs: pairing self-supervised feature representation models like V-JEPA with geometric physical priors, explicit state memory buffers, and real-time physical verification loops. Until then, world models remain effective predictive engines—capable of generating rich virtual environments—yet strictly bounded by the gap between statistical correlation and physical reality.
