What separates a manipulation dataset that works from one that doesn't

Key takeaways

Two manipulation datasets with identical episode counts can produce very different policies, and the difference is set by design decisions made before collection starts.

Condition diversity is the strongest single predictor of transfer, and it cannot be added to a finished dataset.

Failure episodes, intent annotation, and per-episode provenance cost almost nothing during collection and are expensive or impossible to reconstruct afterward.

Language annotation is the one property on this list that can be repaired retroactively, with documented gains from relabeling alone.

Run an acceptance test on a dataset before training on it, using coverage against the protocol and agreement among annotators as the two headline numbers.

You can describe a manipulation dataset accurately with episode count, task list, and hours of robot time, and still have no idea whether a policy trained on it will work outside the room where it was collected.

Those headline numbers say what the collection cost. They say almost nothing about what it teaches, in the same way that dataset size says little about whether you have usable ground truth. Two datasets with the same numbers can differ completely in generalization, and the divergence traces back to five decisions, most of them made before anyone starts recording.

This is a checklist for making those decisions deliberately, and for auditing a dataset you have already been handed.

Decision one: what varies during collection

This is the highest-leverage decision available, and the evidence for it is unusually direct. SEVO (arXiv:2605.11114) found that deliberately varying lighting, backgrounds, and distractors during teleoperation was the single most important factor for generalization, while data collected only in-distribution produced near-zero transfer to new environments.

Near-zero is the part worth sitting with. A dataset collected in one room, under one lighting setup, with one set of object instances is not a weaker version of a diverse dataset. For transfer purposes it can be close to worthless, regardless of how many episodes it contains.

What to specify before collection:

Lighting conditions and how many episodes per condition

Background and surface variation

Object instances per object class, not just object classes

Operators, since teleoperation style is itself a variable

Starting states and approach angles

Distractor objects present in the scene

The number that matters is instances per condition, not total episodes. A thousand episodes across four conditions teaches less than four hundred across twenty.

This decision is unrecoverable. There is no post-processing step that adds lighting diversity to footage shot under one lamp.

Decision two: whether failures are kept

Collection produces failures continuously. The default pipeline trains on successful demonstrations and deletes the rest, which discards the signal that tells a policy how to recover.

VINE (arXiv:2512.03913) describes those failures as encoding where and how policies are fragile. Without them, minor execution errors compound into unrecoverable out-of-distribution states (arXiv:2605.08434), because the policy has never observed a recovery.

The demand is visible in what teams now build specifically to fill the hole: RoboFAC with 9,440 erroneous trajectories and 78,623 question-answer pairs (arXiv:2505.12224), FailSafe pairing failures with executable recovery actions (arXiv:2510.01642), and ViFailback with 58,000 failure-diagnosis pairs (arXiv:2512.02787). These land as headline contributions because success-only collection is the norm.

Keeping failures requires a schema decision and an annotator who can say what went wrong, at which point, and whether the recovery worked. Both are cheap while the session is running.

Decision three: whether intent is recorded

A trajectory records what the arm did. It does not record what the operator was trying to do, which sub-goal was in progress, or why a particular grasp was chosen over an alternative.

That information exists only in the operator's head at collection time, and reconstructing it later from video is guesswork. Recording it is a workflow question: a short structured annotation after each episode, captured while the operator still remembers, costs seconds and produces the labels that make a trajectory interpretable.

This is where domain understanding stops being optional. Someone who does not understand the task cannot annotate intent, which is a different requirement from the throughput-oriented annotation most pipelines are built around. We looked at how the annotator profile changes with task difficulty separately.

Decision four: how language is attached

Language annotation is the exception on this list, because it is the one property you can genuinely fix after the fact.

The default is bad. Instructions in current robot datasets are repetitive, template-like commands with limited structural variation (arXiv:2601.03136), producing what one paper calls modality imbalance (arXiv:2512.11218) and another calls information collapse, where the instruction becomes predictable from the visual scene and stops carrying information (arXiv:2601.15197). Models respond by ignoring language entirely; LIBERO-PRO (arXiv:2510.03827) found outputs unchanged when instructions were replaced with meaningless tokens.

The measured upside of fixing it is large. LangGap (arXiv:2603.00592) moved single-task success from 0% to 90% through targeted, diverse instruction augmentation. CAST (arXiv:2508.13446) added 27 percentage points on navigation via counterfactual relabeling, with no new collection. If you have an existing dataset with thin language, this is the highest return available per dollar.

Decision five: whether provenance survives

Per-episode provenance means knowing, for each trajectory, which robot produced it, which operator, under which conditions, on which date, with which software version.

Two reasons to keep it. The first is that pooling matters: a 2026 scaling study found that combining heterogeneous robot datasets often induces negative transfer, where extra data from other embodiments degrades performance (arXiv:2602.09722). Diagnosing that requires knowing which episodes came from where. Without provenance, a mixed corpus is an undebuggable blob.

The second is security. A demonstrated data-poisoning attack achieved 98 to 99% backdoor success by corrupting 0.31% of training episodes (arXiv:2510.10932). At that ratio, source-level verification is a control rather than a nicety, and it requires provenance to be usable.

Auditing a dataset you already have

If a dataset arrives without this history, you can still test it before committing training time.

Count instances per condition

Not episodes, instances. Group the data by the conditions you care about at deployment and look at the thinnest cell. That cell sets your transfer expectations.

Corrupt the language and retrain a small model

Replace instructions with random tokens on a subset and see whether behavior changes. If it does not, your language channel is carrying no information and relabeling is the cheapest available intervention.

Check the failure fraction

If the dataset is close to 100% successful episodes, it was filtered, and the corrective signal is gone. Ask whether the failures were retained anywhere before assuming they are unrecoverable.

Measure agreement on the judgment fields

Any annotation involving intent, failure cause, or quality is a judgment, and judgments need agreement measurement to be trustworthy. Low agreement on a field means that field is either ambiguous or under-specified, and training on it propagates the ambiguity. Running these checks inside automated quality workflows makes the audit repeatable across collection rounds rather than a one-time exercise.

Together these four checks take a few days and routinely change what a team does next, usually by redirecting spend from more collection toward relabeling or targeted diversity.

Design the dataset, then fund the collection

Four of the five decisions above are unrecoverable once the rig comes down, which makes the collection spec more consequential than the collection schedule. If you are planning a manipulation dataset, or you have inherited one that underperforms, HumanSignal Services designs the protocol, runs the operation, and delivers a dataset built to spec with the quality workflow attached. Book a scoping call to work through the specification.