Why robot foundation models are starved for the right data

Key takeaways

Robot foundation models are not primarily short on trajectory volume. They are short on four properties: condition diversity, grounded language, failure coverage, and sensing beyond RGB.

Adding more data of the wrong kind can reduce performance. Pooling heterogeneous robot datasets has been shown to induce negative transfer, where extra data from other embodiments makes a model worse.

A curated 5% subset has recovered 85 to 90% of full-dataset performance, which implies most of the volume in current corpora carries little signal.

Each of the four missing properties is an annotation and protocol decision, not a collection-hours decision, and three of the four are unrecoverable once collection ends.

Before buying more trajectories, specify what "right" means for your deployment and measure the dataset you already have against it.

Every team building a robot foundation model eventually hits the same wall, and the first instinct is almost always to buy more demonstrations. That instinct is reasonable, and the evidence increasingly says it is wrong.

The problem is not that the corpus is small. It is that the corpus is uniform in the dimensions that matter for transfer, thin in the annotations that make a trajectory interpretable, and missing entire sensing modalities. This piece breaks "the right data" into four properties you can specify and measure, with what the literature says each one is worth.

More data can make the model worse

Start with the finding that breaks the buy-more-data instinct. A 2026 study on rethinking scaling (arXiv:2602.09722) found that pooling heterogeneous robot datasets often induces "negative transfer": adding trajectories from other robot embodiments degraded performance rather than improving it.

That result pairs with a data-distillation finding (arXiv:2511.16233) in which a curated 5% coreset recovered 85 to 90% of full-dataset performance. Read together, they say the same thing from opposite directions. Most of the volume in current corpora is doing very little work, and some of it is doing negative work.

MPVI (arXiv:2606.00985) adds the deployment version of the observation, documenting long-horizon failures that persist despite finetuning on large teleoperated datasets. Scale helps build broad pretraining priors. For a specific capability you intend to ship, composition decides the outcome.

Property one: condition diversity

The most direct evidence for what "right" means comes from SEVO (arXiv:2605.11114), which found that deliberately varying lighting, backgrounds, and distractors during teleoperation was the single most important factor for generalization. Data collected only in-distribution produced near-zero transfer to new environments.

This is a protocol decision, made before collection, and it cannot be repaired afterward. A thousand hours recorded in one room under one lighting setup does not become diverse through post-processing. If you are specifying a collection today, the list of conditions you intend to vary, and the number of instances per condition, is the highest-leverage document in the project.

Property two: language that carries information

Vision-language-action models are named for a language channel that, in practice, many of them ignore.

The literature diagnoses this consistently. Instructions in today's robot datasets are repetitive, template-like commands with limited structural variation (arXiv:2601.03136). One paper terms the result "modality imbalance," where language diversity falls far below visual and action diversity and biases models toward visual shortcuts (arXiv:2512.11218). Another calls it "information collapse": because the instruction is predictable from the visual scene alone, the mutual information between instruction and action approaches zero (arXiv:2601.15197).

The consequence is measurable. LIBERO-PRO (arXiv:2510.03827) showed model outputs holding steady even when researchers corrupted the instructions or replaced them with meaningless tokens. So is the fix. LangGap (arXiv:2603.00592) moved single-task success from 0% to 90% with targeted, diverse instruction augmentation, and CAST (arXiv:2508.13446) added 27 percentage points on navigation through counterfactual language relabeling without collecting a single new trajectory. We covered why this happens and what the relabeling looks like in more detail separately.

Relabeling existing trajectories with diverse, fine-grained language remains one of the cheapest interventions available, and one of the few on this list that works retroactively.

Property three: failure coverage

Collection sessions produce failures continuously. Slips, collisions, mis-grasps, and near-misses happen on the way to every successful demonstration. The field's norm is to train on the successes and discard the rest.

As VINE (arXiv:2512.03913) puts it, those discarded failures encode where and how policies are fragile. Without them the model has no corrective signal, and minor execution errors compound into unrecoverable out-of-distribution states (arXiv:2605.08434).

You can see the demand in the datasets now being built to supply what the norm throws away: RoboFAC ships 9,440 erroneous trajectories with 78,623 question-answer pairs (arXiv:2505.12224), FailSafe pairs failures with executable recovery actions (arXiv:2510.01642), and ViFailback provides 58,000 failure-diagnosis pairs (arXiv:2512.02787). Each lands as a headline contribution because the data does not otherwise exist.

Keeping failures costs almost nothing at collection time. It costs a schema decision and an annotator who can say what went wrong and when. Recovering them later costs a new collection campaign.

Property four: sensing beyond RGB

The corpus is overwhelmingly RGB plus language plus proprioception. Tactile and force sensing, both essential for contact-rich manipulation, are close to absent. In our analysis of the vision-language-action literature, one full chunk of 123 papers contained zero tactile-centric work, and audio appeared in three efforts across the whole 1,228-paper corpus.

Papers that do tackle these modalities name the same root cause: scarce aligned vision-tactile-language data (arXiv:2605.27886) and the absence of large multimodal datasets (arXiv:2507.17294). The datasets that exist, such as HapTile, ForceVLA, and OmniVTLA, are flagged as flagship contributions for the plain reason that nothing comparable exists.

If your deployment involves insertion, deformable objects, or anything where vision cannot tell you whether contact succeeded, this gap is your gap, and closing it means instrumenting collection rather than filtering an existing corpus.

Uncurated data is also an attack surface

One finding deserves separate mention because it changes the risk calculation rather than the performance calculation. Two 2025 papers demonstrated data-poisoning attacks on vision-language-action training, one achieving 98 to 99% backdoor success by corrupting just 0.31% of training episodes (arXiv:2510.10932).

At that ratio, provenance and verification stop being data-hygiene concerns and become security controls. A corpus assembled from mixed sources without per-source verification is difficult to audit after the fact.

What to do before buying more trajectories

Write down what "right" means for the capability you are shipping, then measure what you already have against it. Four questions cover most of the ground.

Which conditions vary in your current data, and how many distinct instances exist per condition?

Are your language annotations predictable from the scene alone, and what happens to model behavior when you corrupt them?

What fraction of your episodes are failures, and are they annotated with what went wrong?

Which sensing modalities does your task depend on, and which of those are actually recorded?

Two of these you can answer today from the dataset you have. The language question you can answer with an afternoon's experiment. The modality question usually decides whether the next collection round needs new hardware. Teams that answer all four before scaling collection tend to spend less and ship sooner, because the spend is aimed at a named deficiency instead of at volume. Running that audit inside a workflow that tracks agreement and coverage per field makes the answers repeatable rather than one-off.

Our broader read of the vision-language-action literature covers how the field arrived at this position, and the rest of our physical AI research takes each of these properties in turn.

Specify the dataset before you fund the collection

The four properties above are collection-protocol decisions, and three of them are unrecoverable once the rig comes down. If you are planning a collection round for a robot foundation model, the spec is worth more scrutiny than the schedule. HumanSignal Services designs the protocol, runs the operation, and delivers a verified dataset built to that spec. Book a scoping call to walk through what your capability actually requires.