VLA data vs. teleoperation data: what's the difference?

A team books three months of teleoperation, records forty thousand episodes, and then discovers the data will not train the policy they had in mind. Nothing failed. The rig worked, the operators were good, the files are intact. The problem is that teleoperation describes how data was produced and a vision-language-action model describes what a policy consumes, and the two are separated by work that mostly has to happen while the rig is still standing. Treating them as the same thing is one of the more expensive category errors in robotics data, and it is easy to make because every vendor quote blurs them.

Key takeaways

• Teleoperation is how data gets made; VLA data is what a policy consumes. A teleoperation dataset is an input to VLA data, not a synonym for it.
• The distance between them is language grounded to the right segment, task and subtask boundaries, labeled failure and recovery, and conditions varied enough to generalize.
• DROID makes the cost visible. It published 76,000 successful episodes, and its own policy experiments used the first 40,000 for which language annotations existed at the time.
• Most of those additions are cheap while the rig is up and expensive or impossible afterward, which puts them in the collection protocol rather than a later annotation phase.
• Teleoperation is not the only route. EgoDex, an egocentric human capture set, produced 829 hours and 338,000 episodes across 194 tasks without a robot in the loop.

One is a method, the other is a format

Teleoperation is a collection technique. A human operates a robot through a leader device, VR controllers, or a motion-capture rig, and the system records what the operator commanded alongside what the robot and its sensors observed. The output is a set of synchronized observation and action streams.

VLA data is a description of what a vision-language-action policy needs at training time: observations, actions, and language that refers to what is happening, aligned closely enough that the model can learn the correspondence between an instruction and the behaviour that satisfies it.

The overlap is real, which is why the confusion persists. Teleoperation is the most common way to produce the observation and action halves. It does not, on its own, produce the language half or the structure that makes the correspondence learnable, and that is the part teams discover late.

What teleoperation produces

A well-run session yields, per episode, a time series of robot states and commanded actions, one or more synchronized camera streams, and whatever additional sensing the rig carried. Metadata should include calibration, camera intrinsics, and measured per-step timing, though whether it does is a separate question covered in the metadata failures that ruin a recorded session.

That list is worth reading as what a recording is good at. Teleoperation captures the physical correspondence between intent and outcome with high fidelity, at whatever rate the rig sustained, in the exact conditions the session ran under. Nothing else produces that correspondence as directly, which is why it remains the default despite the cost.

What arrives is a faithful recording of a person doing a task through a machine. What does not arrive is any statement of what the task was, where one sub-behaviour ended and the next began, whether the attempt succeeded, or why it failed when it did.

What a VLA model needs on top of that

Four additions close the distance, and they are the substance of what a vision-language-action policy needs from its data.

Language grounded to the segment it describes

An episode-level caption is nearly free and teaches little. What a policy can learn from is language attached to the span of time it refers to, so that "grasp the handle" is aligned with the frames in which the grasp occurs. Producing that alignment after the fact means someone rewatching footage and marking boundaries, and the quality of the result depends on how well they can infer intent they did not witness. Whether existing grounding holds up is checkable, which is the point of auditing the language field on a dataset you already have.

Task and subtask boundaries

Policies that reason about multi-step behaviour need to know where steps begin and end. Left unsegmented, a long episode presents as one undifferentiated demonstration, and structure a model could have exploited is unavailable. The boundaries are cheapest to record at collection, when the operator knows what they were doing.

Failure and recovery, labeled as such

A dataset of clean successes teaches a policy nothing about what going wrong looks like or how to get out of it. Failures need to be retained with the cause attached rather than discarded, which is a protocol decision rather than an annotation one, since it determines whether the episode is kept at all.

Conditions varied enough to generalize

A policy trained on one lighting setup in one room learns that room. Condition diversity is among the properties these models are short on, and unlike a label it cannot be added later at any price.

Why the difference shows up as cost

The clearest public illustration comes from DROID, which reports both sides. It published 76,000 successful episodes, and for its own policy learning experiments used the first 40,000 successful trajectories for which language annotations were available at the time of training. That was an experimental choice rather than a verdict on the remainder, and the shape of it is the lesson: collection had run well ahead of annotation, and annotation was what bounded the training run. Read alongside the fact that a substantial share of a campaign never survives curation, the gap between episodes recorded and episodes usable turns out to be where most of the budget goes.

The asymmetry is what makes this worth planning around. Language, segmentation, and failure causes can technically be added later, at the price of someone reconstructing intent from video. Condition diversity, calibration, and sensing that was never captured cannot be added at all. Sequencing a project as "collect now, annotate later" therefore trades a cheap task for an expensive one and forecloses several others entirely.

Other routes to VLA data that are not teleoperation

Teleoperation is the default, not the only option. Egocentric human capture records a person performing tasks directly, with EgoDex gathering 829 hours and 338,000 episodes across 194 tasks with paired 3D hand and finger tracking, no robot involved. It scales more easily and carries no robot embodiment, which is a benefit or a problem depending on whether your policy needs to transfer across morphology. Practical considerations are covered in capture without a robot in the loop.

Simulation and scripted play produce large volumes cheaply with perfect labels and a reality gap that has to be managed. None of these is strictly better; they differ in what they make easy, which is the same reasoning behind the other routes teams are using and behind the data problem underneath these models generally.

How to specify a campaign that produces both

Write the training format into the collection protocol rather than treating annotation as a downstream phase. In practice that means deciding the task taxonomy and segment boundaries before the first session, having operators state or mark the instruction per segment while they still remember it, recording failure cause as a required field, setting condition quotas across lighting, scene, and object variation, and asserting that every field your training pipeline reads is present and non-default on a trial episode before production recording starts.

None of this is expensive during collection. All of it is expensive or impossible afterward.

The organizational version of the same point is that annotation capacity has to be sized against collection throughput from the start, rather than added once a backlog is visible. A campaign that records faster than it annotates does not save money; it converts a cheap in-session task into an expensive reconstruction task and defers the moment anyone discovers the data will not train what they wanted.

Specify the campaign against the format you need

The distance between a teleoperation session and a trainable VLA dataset is protocol work, and protocol work is cheapest before the rig goes up. HumanSignal Services designs collection protocols, runs the operation, and delivers episodes with grounded language, segmentation, and failure annotation recorded as data rather than reconstructed from video. Book a scoping conversation to work through what your policy needs from a campaign.