Why one in six of your teleoperation episodes gets thrown away

You budgeted a collection campaign in sessions: so many operators, so many hours, so many episodes. The dataset that reaches a training run is smaller than that number, often much smaller, and the shrinkage happens in three separate places that nobody costed. Some episodes fail during collection. Some survive collection but arrive without the annotation or calibration a training pipeline needs. The rest face a curation step that deliberately throws away more. Understanding where each loss occurs turns an unpleasant surprise into a planning input, and the published evidence suggests the last of the three is doing you a favour.

Key takeaways

• DROID published 76,000 successful episodes and recorded roughly 16,000 more as unsuccessful, which is about one in six of everything collected, set aside before the dataset was released.
• The larger loss comes later. DROID's own policy experiments trained on the first 40,000 successful trajectories that had language annotations, so well under half of what was collected reached that training run.
• Discarding is not waste. On manipulation benchmarks, smoothness-based curation reached 16% higher success using one sixth of the data, and a second metric gave a 20% improvement on half.
• That makes the discard rate a design parameter. Budget collection by expected yield after curation rather than by session count.
• The losses you cannot recover are the metadata ones. A failed episode still carries signal; an episode with no calibration or timing record carries none.

Where the losses occur

Three stages, each with a different cause and a different remedy.

At collection: the episode failed

The operator knocked the object over, the grasp slipped, the arm hit a joint limit, or the task was simply not completed. These are ordinary and expected, and they are the losses teams do anticipate, though usually at a rate lower than they turn out to be.

At publication: the episode is unlabeled or uncalibrated

An episode can succeed and still be unusable because something the training pipeline requires is missing: a language annotation, camera intrinsics, a calibration snapshot, or reliable per-step timing. This stage is the quiet one, and it is the only stage where the loss is avoidable at zero marginal cost. It is also where defects in the episodes that do survive originate, since an episode with wrong metadata is worse than one with missing metadata.

At training: the episode did not make the curated subset

The third stage is deliberate. Teams filter for quality, balance, or task coverage, and the filter is often aggressive. Unlike the first two, this loss is a choice, and the evidence says it is frequently the right one.

The three stages compound multiplicatively rather than adding up, which is why the final number surprises people. An 83% collection success rate, a 90% metadata completeness rate, and a curation step keeping half leaves you with roughly 37% of the episodes you paid to record. Each individual figure sounds acceptable in isolation. Their product is what determines whether the campaign hits its target.

The numbers from a dataset that documented them

Most robotics datasets report what they contain rather than what they discarded, which makes DROID unusually useful because it reports both. The paper states that DROID consists of 76,000 successful episodes, and that roughly 16,000 trajectories in the collection were labeled "not successful," released alongside the dataset but not counted toward its size.

Work the arithmetic through. Roughly 92,000 episodes were collected, 16,000 of them were set aside, and that is about 17%, or close to one in six. Those episodes were gathered across 564 scenes in 52 buildings, covering 86 tasks, by 50 collectors over twelve months, so the rate reflects sustained collection across varied conditions rather than one team's bad week.

The second loss is larger and easier to miss. For their policy learning experiments the authors used the first 40,000 successful trajectories for which language annotations were available at the time of training. That is an experimental choice rather than a statement that the remaining episodes are unusable, and the arithmetic is still instructive: of roughly 92,000 episodes collected, about 40,000 fed that particular training run. The annotation backlog, not the failure rate, was the binding constraint.

This is the collection-side counterpart to the argument that a curated subset recovers most of the performance of a full corpus. If most of the volume carries little signal, a large discard is not evidence of a broken process.

Why discarding more can improve the policy

Recent work on demonstration curation makes that concrete. Scoring trajectories by smoothness, using spectral arc length, and keeping only the best, one method reached 16% higher success on the RoboMimic benchmarks while training on one sixth of the data. A second metric based on trajectory-envelope distance produced a 20% improvement on real-world manipulation using half the demonstrations. Used instead as a retrieval-stage re-ranking filter within an existing pipeline on LIBERO-10, the same scoring gave a 5.6 percentage point mean improvement in success.

Read those results together and the picture inverts. More demonstrations of uneven quality can be worse than fewer clean ones, because an imitation policy fits whatever it is shown, including hesitation, correction, and jerky recovery. That connects directly to the question of how many demonstrations a policy needs: the answer depends on what fraction survives curation, which depends on how the sessions were run.

What this means for how you budget collection

Plan the yield, not the session count

Work backward from the training set you need. If you want 40,000 usable trajectories and expect roughly 17% collection failure plus a curation step keeping the better half, you need to collect on the order of 96,000 episodes, not 40,000. Teams that budget by session count discover the shortfall at the end, when re-running collection costs the most because the rig, the operators, and the scene setups have been released.

Instrument so a failed episode is still informative

A failed episode is not a wasted one if you recorded why it failed. Capture the failure mode, the phase of the task at which it occurred, and the full sensor stream. Those episodes support failure-recovery training and populate the failure coverage that policies trained only on clean successes conspicuously lack, which is part of what separates a manipulation dataset that works from one that merely looks complete.

Decide the curation metric before collection ends

If you intend to filter on smoothness, task-phase coverage, or contact quality, decide that while the rig is still up. Some curation metrics need signals that have to be recorded during collection, and a metric you adopt afterward can only use what you happened to capture. This is the same reasoning that governs where teams are sourcing trajectories in the first place: sourcing and curation are one decision, not two.

When a failed episode is worth keeping

Keep failures when your policy needs to recognize or recover from them, when the failure mode is one you expect in deployment, and when the episode is complete enough to show what went wrong. Some recent large manipulation datasets now retain failed trajectories deliberately, labeled with a cause rather than filtered out, which is a reasonable default for any collection where deployment conditions are messier than the lab.

Discard, or at least quarantine, episodes whose metadata is unreliable. An episode with drifted calibration or misreported timing does not merely add noise; it teaches a relationship between action and outcome that did not occur. A clean failure is training signal, and a well-executed success with wrong metadata is a corrupted label.

The asymmetry is worth stating plainly, because it inverts the intuition most teams bring to data cleaning. Failure is a property of the episode and is recoverable as information, since you still know what the operator did and what the world did in response. Bad metadata is a property of the record and destroys that correspondence, and no downstream process can reconstruct it. Teams routinely spend effort reducing the failure rate, which is the loss that carries residual value, while tolerating the metadata loss, which does not.

Plan a collection against the yield you need

The discard rate is knowable in advance, and it changes how many sessions you book, how you instrument them, and what you record per episode. HumanSignal Services designs collection protocols, runs the operation, and delivers datasets with calibration, timing, and failure annotation recorded as data rather than reconstructed afterward. Book a scoping conversation to work through the yield your target dataset implies, or read how physical AI collection is run end to end.