Why world models can't be trained on scraped video alone

Key takeaways

Internet video is effectively unlimited, and it is action-free: it records what happened without the action that caused it, which is exactly what a controllable world model needs.

Action labels are described in the literature as rarely available and costly to obtain, which makes the missing channel a labeling problem rather than a supply problem.

Two escape routes exist. Inverse-dynamics bootstrapping used roughly 2,000 human-labeled hours to unlock around 70,000 machine-labeled hours, and latent action models reached better results than a ground-truth baseline at over 30 times the efficiency.

Both routes still need a human-verified seed. Latent action learning provably requires supervision when distractors are present, and one leading method needs about 150 ground-truth trajectories per task.

Scraped video also needs heavy curation before it teaches anything, with major corpora keeping as little as 10% of what they ingest.

The appeal of training world models on internet video is obvious. The supply is effectively unlimited, it costs nothing to acquire, and it depicts the physical world in more variety than any collection campaign could produce.

Then you try to make the model controllable, and the problem surfaces. A world model is meant to answer what happens if I do this. Video answers what happened. The link between the two, the action, was never recorded.

This piece walks through what that missing channel costs, the two ways the field works around it, and why both still depend on a small amount of human-verified data.

The missing channel

Every frame pair in a video implies a transition. What produced the transition is absent: the command, the control input, the intent, the force applied.

For a passive predictive model that is tolerable. For a model you intend to condition on actions, it is the whole problem. Genie's authors put it directly, describing action labels as rarely available and costly to obtain (arXiv:2402.15391), and the same framing recurs across the latent-action literature (arXiv:2410.11758, arXiv:2312.10812).

This is worth naming precisely, because it is easy to mistake for a scale problem. There is no quantity of additional video that supplies the action channel. Ten million more hours of unlabeled footage leaves the model exactly as uncontrollable as before. The constraint is structural.

The robotics field arrived at the same wall from the opposite side. There, the actions exist because a robot executed them, and the bottleneck is that collecting trajectories on hardware is costly and difficult to scale, which is the finding that runs through our analysis of 1,228 vision-language-action papers. One field has the video and lacks the actions; the other has the actions and cannot get enough video. Both are short the pairing.

Route one: infer the actions

The first workaround trains a model to recover actions from observed transitions, then uses it to label the unlabeled corpus.

Video PreTraining is the canonical demonstration. Roughly 2,000 hours of human-labeled gameplay trained an inverse-dynamics model, which then labeled around 70,000 hours of unlabeled footage (arXiv:2206.11795). The ratio is the point: a bounded, affordable human effort unlocked a corpus 35 times larger.

The dependency is equally clear. The inverse-dynamics model is only as good as its seed set, and errors in that seed propagate silently through every machine-labeled hour that follows. There is no downstream signal that catches a systematically mislabeled action, because the labels are the ground truth by construction.

That makes seed quality disproportionately important. A small set that is carefully collected, reviewed, and checked for agreement among the people producing it is worth considerably more than a larger one assembled quickly.

Route two: learn a latent action space

The second workaround declines to name actions at all. Latent action models learn a compact representation of what changed between frames and condition generation on that, without explicit labels.

The results are strong. LAPA outperformed a ground-truth-action baseline by 6.22% while being more than 30 times more efficient to train (arXiv:2410.11758). For a method that never sees an action label, that is a striking outcome.

The failure mode is where the practical caution lives, and it is documented carefully. Latent action models encode exogenous information, meaning they absorb changes in the scene that had nothing to do with the agent's action. Distractors derail them, and there is a formal result showing that latent action learning provably requires supervision in the presence of distractors (arXiv:2605.20223, arXiv:2502.00379). Later work continues to probe where the representation breaks (arXiv:2511.16407).

LAPA itself still needs roughly 150 ground-truth trajectories per task to ground its latent space, and it stumbles on rare events such as grasping. The efficiency gain is real, and it is a gain over full labeling rather than over labeling entirely.

The reason this matters operationally is that the failures are quiet. A latent action model that has absorbed a distractor does not error out. It produces plausible predictions conditioned on the wrong thing, which surfaces only when the model is deployed and behaves inexplicably.

Even the video needs curating

Suppose the action channel were solved. Scraped video would still not be ready to train on.

The large corpora discard most of what they ingest, and they report that this improves results rather than only reducing cost. Cosmos removes the bottom 15% on quality and roughly 30% more through deduplication, stating that much raw video does not contain useful information for learning the physics of the world (arXiv:2501.03575). Genie retained about 10% of its raw hours, and the curated subset beat the full set on video quality metrics (arXiv:2402.15391). Koala-36M filtered a 70 million clip pool to 36 million using a single learned suitability score (arXiv:2410.08260).

The filters encode judgments about what counts as physically informative footage, and those judgments were made by people and then automated. Volume ingested tells you about bandwidth. What survives curation tells you about the training set.

What the pairing actually requires

The practical shape of a world-model data program follows from the two routes above.

You need a seed set of action-paired data, collected under a protocol, covering the interactions your model must control, with the action recorded alongside the observation rather than inferred later. It is small relative to the corpus and it determines the ceiling of everything built on it.

You need curation criteria applied to the unlabeled corpus, defined by someone who can say what physically informative footage looks like for your domain and then encoded into a filter.

You need a verification step for whatever the automated labeling produces, because both routes fail quietly. Sampling machine-labeled segments back to human review is the only way an inverse-dynamics drift or a latent-action distractor gets caught before training.

And you need coverage of the cases the scraped distribution under-represents. Rare events are sparse in logged data by definition, which is why collisions and near-misses are described as exceedingly sparse in driving logs (arXiv:2501.11260), and why models trained on that data are weakest exactly where the stakes are highest.

That is a data-collection program with a protocol and a review workflow, sitting alongside the scraping. It is smaller than the corpus and it is the part that decides whether the corpus is usable. Reviewing paired video and action streams also needs tooling that can show several synchronized streams together, since a misaligned action label is indistinguishable from a wrong one.

Build the seed set deliberately

The action-paired seed, the curation criteria, and the verification pass are the three pieces scraping cannot provide, and all three set the ceiling on everything trained afterward. HumanSignal Services designs those collections, runs them, and delivers verified datasets built to your spec, including purpose-built egocentric capture. Book a scoping call to work through what your model would need.