What data do you need to train a VLA model?

More data built the large language models. For Vision-Language-Action (VLA) models, that assumption breaks. Adding robot demonstration data from different platforms and embodiments can actively degrade performance on your target task. The binding constraint isn't how many episodes you've collected. It's what those episodes contain, how varied they are, and how faithfully their language annotations describe the actions they show.

TL;DR

A VLA model trains on three distinct data types: visual observations, language instructions, and action trajectories.

Pooling heterogeneous robot data causes negative transfer, where added data hurts performance on your specific task.

Repetitive language annotations let models ignore the language channel entirely.

Visual diversity across lighting and object arrangements predicts generalization better than episode count.

Audit your existing data variety before collecting more demonstrations.

What a VLA model is (and why its training data differs from standard vision models)

Standard vision models train on static images paired with category labels, such as labeling a coffee cup image as "coffee cup." This relationship is fixed and context-free.

A VLA model works differently. It integrates visual perception, natural language understanding, and physical action generation into a single framework (Cornell and HKUST). The model takes RGB or depth camera observations and a language instruction as input. Output is direct robot control: joint angles, motor torques, or end-effector positions.

The action output changes what data collection requires. A motor command sequence is time-ordered and only meaningful in relation to the visual state and language instruction that generated it. You can't label a trajectory the way you label an image. You have to capture what the robot saw, what instruction it followed, and every motor command it issued, in sequence.

Internet-scale pretraining on vision-language data is the defining characteristic of a true VLA (Reuss, ICLR 2026). It gives these models stronger language-instruction following than systems trained only on robot demonstrations. That pretraining is table stakes. The data decisions that determine whether a VLA works in production happen at fine-tuning.

The three inputs a VLA model learns from

Visual observations

Visual observations are camera feeds: RGB images, depth maps, and similar inputs. The collection challenge is diversity, not resolution or frame rate. A model trained on demonstrations where the object always appears centered, against a clean background, under consistent lighting, learns those regularities. Shift any one of them in production and performance drops.

Useful visual data covers the conditions your production environment will present: different lighting, cluttered backgrounds, and objects at varying orientations. The specific conditions matter less than ensuring you vary them.

Natural language instructions

Language instructions are the most commonly underbuilt part of a VLA dataset. If every demonstration of "pick up the cup" uses a single cup in an otherwise empty scene, the instruction is redundant. The model predicts "move to the cup and grasp" from vision alone, without processing the language at all.

Useful language data requires variety. Include scenes where two objects are present and the instruction is the only signal distinguishing which to manipulate. Use instructions that describe the same action at different specificity levels: "move the red mug to the left shelf" versus "put the mug away."

Action trajectories

Action trajectories are the time-stamped sequences of motor commands that constitute the robot's ground-truth behavior. VLA models differ in how they represent this output (survey of architectures). Autoregression-based models treat actions as discrete tokens. Diffusion-based models generate continuous trajectory distributions. Hybrid models combine transformer backbones with dedicated action decoders.

How a model represents actions affects data formatting, but collection challenges are consistent across all three approaches. Each trajectory must be paired with the exact visual state and instruction that generated it. A trajectory paired with an inaccurate or ambiguous instruction is worse than no data at all.

OpenVLA trained on over 1 million episodes from 22 different robot embodiments. The 7-billion-parameter model outperformed a 55-billion-parameter closed model by 16.5 percentage points across 29 tasks. Scale clearly matters, but it also reveals why cross-embodiment pooling creates its own problems.

Why pooling more robot data can hurt performance

Negative transfer

Data, not architecture or compute, is the binding constraint on VLA progress (2026 VLA data analysis). Naive scaling by pooling heterogeneous robot data induces negative transfer. Demonstrations from a different platform, gripper, or task domain introduce conflicting action distributions. The model learns a blurred average that performs worse on the behavior you need.

This happens because a pick-and-place trajectory for a parallel-jaw gripper looks different in joint space than the same task performed by a suction gripper. Pool those trajectories without weighting or filtering, and the model's action distribution shifts away from either. Any robot using that model inherits the confusion at inference time.

Information collapse

When language annotations are repetitive, models stop using language as an input signal (arXiv:2601.15197). If "pick up the cup" appears in 10,000 episodes and the cup is the only graspable object in 9,800 of them, the model predicts the grasp from vision alone. Language becomes noise. The model filters it out.

When vision dominates the prediction signal, the model discards the language channel during inference (arXiv:2512.11218). The result is a robot that can't follow instructions requiring language to disambiguate. Move the cup next to a bottle, and the model has no mechanism to resolve which object the instruction names.

The "more is better" logic isn't wrong for pretraining the vision-language backbone. Internet-scale image-text data does improve instruction following. The problem is at the action fine-tuning stage, where cross-embodiment demonstration data has conflicting action distributions. Applying LLM-era scaling to robot demo collection is where these failures emerge.

The annotation challenge: labeling robot demonstration data

What reviewers actually see

When you annotate images, you look at a frame and apply a label. For episode annotation, you watch a motor command sequence play out and verify that the language instruction describes what the robot did. That includes catching cases where the trajectory was correct but the instruction was wrong.

That last category is particularly difficult to spot. An episode where the robot picks up the red cup while the instruction says "pick up the cup" looks fine. It fails when your environment has two cups. Catching that ambiguity prevents a training example that teaches the model to ignore color as a disambiguating signal.

To review an episode, you need the visual state at the start, a replay of the robot's trajectory, and the associated language instruction. You also need enough frame-level detail to spot where the action diverged from what the instruction specified. Automated tools can pre-label visual frames and flag low-confidence predictions. They can't explain why a specific language-action pairing is wrong.

How interface design affects annotation consistency

The interface a reviewer works in is a direct variable in annotation throughput and consistency. A reviewer clicking through eight screens to reject an episode will catch fewer problems than one who flags and reclassifies in two clicks.

One enterprise team re-architected their evaluation interface using Label Studio Enterprise and cut clicks by 90 percent per annotation task. On a corpus of tens of thousands of robot episodes, that difference compounds into weeks of reviewer time.

Automated metrics surface the candidates most likely to contain problems. Human reviewers explain what failed and why. A score that says "this episode is low confidence" is a starting point. A reviewer who watches the replay and notes "wrong shelf" is the signal that actually improves training.

Diversity over volume: what a strong VLA training set looks like

Treat diversity as a design specification, not a side effect of collecting enough data.

A VLA training set should address diversity across all three modalities:

Visual variation: Collect demonstrations under at least three lighting conditions (overhead fluorescent, diffuse natural light, low ambient). Vary camera angle across episodes. Place distractors in the scene for at least 30 percent of demonstrations, even when the task doesn't require interacting with them.

Language variation: Write at least five phrasings per instruction type, from object-level ("pick up the mug") to spatially specific ("move the blue mug 10 centimeters left of the coaster"). Include scenes where language is the only signal distinguishing two valid actions.

Action trajectory coverage: Collect multiple demonstrations of the same task with intentionally varied robot paths, approach angles, and grasp strategies. A model trained on a single canonical path learns that path, not the task.

Embodiment metadata: Tag each episode with the robot platform, gripper type, and joint configuration. Those tags let you filter or weight cross-embodiment data instead of pooling it blindly.

Failure cases: Include labeled failure demonstrations: missed grasps, correct paths with wrong objects, and instructions followed literally but to poor effect. Models trained only on successes don't learn to recover from predictable errors.

Diversity across lighting and backgrounds predicts generalization better than raw episode volume (VLA data infrastructure research). A 2026 analysis of 1,228 arXiv papers found VLA publication volume grew 5x year-over-year, while broader robotics research grew 127 percent.

The question to ask before your next data collection run

Adding more demonstrations doesn't fix a dataset that lacks visual variety, linguistic specificity, or embodiment metadata. A smaller, well-composed dataset will outperform a larger, homogeneous one on any task where generalization matters.

Before your next collection run, audit whether your data covers the lighting, object arrangements, and instruction phrasings your robot will face in deployment. If not, what you need is a different composition, not more episodes. Building ground truth datasets with real variety, and review interfaces that catch annotation errors before training, is where the work actually happens. The model learns what you give it.