What data a world model needs to understand the physical world

Key takeaways

Visual realism and physical correctness are separable capabilities, and current models are far better at the first than the second.

On the Physics-IQ benchmark, the model with the highest visual realism score placed last on physical understanding, and physics performance across models topped out at 29.5%.

The dominant visual-quality benchmark correlates 0.28 with physics adherence and 0.69 with visual quality, so leaderboard position is a weak proxy for world understanding.

Four properties are missing from scraped video and have to be supplied deliberately: action conditioning, counterfactual coverage, physically informative content, and verified physics ground truth.

Every credible physics benchmark had to construct a bespoke, human-verified dataset, because no off-the-shelf metric can check whether a generated video obeys physics.

A world model is supposed to do something more demanding than generate convincing video. It is supposed to predict what happens next, which requires representing objects that persist, forces that act, and consequences that follow from actions.

Those are different capabilities, and the evidence says current models have one much more than the other. NVIDIA's Cosmos work states the constraint plainly: data determines the ceiling of an AI model (arXiv:2501.03575). This piece is about what has to be in the data for the second capability to appear.

Looking right and being right come apart

The clearest demonstration comes from Physics-IQ (arXiv:2501.09038), which filmed 396 controlled real-world videos to test whether models predict physical outcomes correctly.

The headline dissociation is stark. Sora posted the best visual-realism score of any model tested at 55.6%, and placed last on physical understanding at 10.0%. Across all models, physics performance topped out at 29.5%. Whatever produces convincing texture and motion is not the same thing that produces correct dynamics.

WorldModelBench (arXiv:2502.20694) quantified the consequence for anyone reading leaderboards: the dominant visual-quality benchmark correlates only 0.28 with physics adherence, against 0.69 with visual quality. A model can climb the visible ranking while getting no better at the thing world models are for, which is the general case of why benchmark choice decides what you learn about a model.

Supporting results agree. VideoPhy-2 (arXiv:2503.06800) reports 22% joint performance and documents conservation-law failures, and PhyGenBench (arXiv:2410.05363) ties the gap between optical and mechanical understanding directly to training-data composition. That last connection is the useful one, because composition is something you can change.

Property one: action conditioning

The single largest structural gap is that video records what happened, not what was commanded.

A controllable world model needs to know which action produced which transition. Internet video contains none of that. Genie's authors describe action labels as rarely available and costly to obtain (arXiv:2402.15391), which is the constraint that shapes the whole field.

Two escape routes exist and both need human-verified anchoring. Inverse-dynamics bootstrapping infers actions from observed transitions, exemplified by VPT, where roughly 2,000 human-labeled hours unlocked around 70,000 machine-labeled hours (arXiv:2206.11795). Latent action models learn an action space without explicit labels, and LAPA beat a ground-truth-action baseline by 6.22% at more than 30 times the efficiency (arXiv:2410.11758).

Both are genuine advances, and neither eliminates the human. LAPA still requires roughly 150 ground-truth trajectories per task to ground its latent space. Latent action learning also has a documented failure mode: these models encode exogenous information and are derailed by distractors, with a formal result showing that latent action learning provably requires supervision in the presence of distractors (arXiv:2605.20223, arXiv:2502.00379). The seed set is small, and it is load-bearing.

Property two: content that is physically informative

Most video is not useful for learning physics, and the large corpora say so through their filtering.

Cosmos cuts the bottom 15% on quality and roughly 30% more in deduplication, on the stated grounds that much raw video does not contain useful information for learning the physics of the world (arXiv:2501.03575). Genie kept about 10% of its raw hours, and the curated set outperformed the full set on video quality metrics (arXiv:2402.15391). Koala-36M filtered a 70 million clip pool down to 36 million using a learned suitability score (arXiv:2410.08260).

The pattern is consistent: aggressive curation improves models rather than merely shrinking them. Total hours ingested is a vanity metric. What matters is the fraction that shows objects interacting under forces in ways a model can learn from, which is a small slice of what cameras record.

Property three: the events that are rare and decisive

Two failure modes trace to the same distributional cause.

Long-horizon drift comes from training on clean context and running on the model's own imperfect output, so errors compound. The literature describes mitigations that alleviate but do not eliminate the effect.

Rare-event blindness is the more consequential one. In driving, collisions and near-misses are exceedingly sparse in real logs (arXiv:2501.11260), so a model trained on logged data is weakest on exactly the interactions that matter. The generative failures that follow have been catalogued as detectable classes, including object permanence violations, morphing, and implausible appearance (arXiv:2405.04180).

Rare events cannot be sampled into a dataset by collecting more of the same distribution, because their rarity is a property of that distribution. They have to be deliberately constructed or deliberately sought, which is a collection-design decision rather than a scale decision.

Property four: verified physics ground truth

The last requirement applies to evaluation, and it is the one that gets discovered late.

Every credible physics benchmark had to build a bespoke, human-verified dataset, because generic metrics and off-the-shelf model judges cannot verify whether physics was obeyed. Physics-IQ filmed 396 controlled real videos (arXiv:2501.09038). WorldScore curated 3,000 test examples with camera-trajectory ground truth (arXiv:2504.00983). WorldSimBench and VideoPhy-2 both trained bespoke human-feedback evaluators (arXiv:2410.18072, arXiv:2503.06800).

Nobody did this because it was convenient. They did it because measuring physical understanding requires knowing the physically correct answer, and that comes from controlled capture and human verification. The same dependency shows up when teams lean on model-based judges for scoring, which is why we looked at what happens to a benchmark when you rely on AI judges rather than human-grounded references.

The practical implication is that the evaluation set is a data-collection project of its own, with the same protocol and verification demands as the training set, and it should be scoped at the same time rather than assembled at the end. The same logic applies to benchmarks that need to evolve as models improve, since a fixed physics benchmark saturates.

A specification, not a scraping target

Put together, a world model that understands the physical world needs training data with actions attached to transitions, content selected for physical informativeness rather than volume, deliberate coverage of rare and decisive events, and a separately constructed, human-verified evaluation set.

None of those four is a property of scraped video. Each is produced by a decision about what to record, what to keep, and what to verify. That is the same conclusion the robotics field reached from a different direction, where our analysis of 1,228 vision-language-action papers found the bottleneck had shifted from data quantity to annotation, curation, and verification. Two fields, different modalities, the same constraint.

Scope the data layer alongside the model

If you are building or evaluating a world model, the action labels, the curation criteria, the rare-event coverage, and the verified evaluation set are all data-collection projects with their own protocols. HumanSignal Services designs those collections, runs them, and delivers verified datasets built to spec, including controlled multimodal capture. Book a scoping call to work through what your model needs.