The data integrity failures that silently ruin a teleoperation dataset

You finish a collection run, the episodes load, the tensors have the shapes you expect, and the replay videos look like the task. Weeks later a policy trained on that data underperforms one trained on a smaller set, and nothing in the dataset explains why. The failures that produce this outcome are rarely dramatic. They are metadata that describes the recording incorrectly, hardware state that drifted mid-session, and annotation fields left empty in a way the training code treats as valid input. Each is cheap to detect while the rig is still assembled, and most are unrecoverable afterward.

Key takeaways

• A teleoperation dataset can be structurally valid and still be unusable. The most common failures produce well-formed files that pass shape checks, load tests, and visual spot-checks.
• Recording strategies in LeRobot have written data at the nominal frame rate multiplied by an interpolation factor while the dataset metadata reported the nominal value, so the timing field did not describe the recording.
• Missing subtask annotations have silently produced all-zero training targets instead of an error, which trains a policy against nothing and reports success.
• Leader and follower calibration drift within a session is still caught by tooling a team has to build, rather than by anything in the dataset format.
• Camera intrinsics, per-step timing, and calibration state are unrecoverable once the rig comes down. Almost everything else can be repaired later.

Three failures that pass every smoke test

The examples below are drawn from open and resolved issues in the LeRobot repository, which is useful precisely because it is the most widely used open collection stack. These are not exotic edge cases found in one lab. They are the reports that surface when many teams run the same tooling against real hardware.

Calibration drift between leader and follower

In a leader-follower teleoperation setup, the mapping between the operator's device and the robot's joints is established at calibration and assumed to hold. It does not always hold. Mechanical settling, thermal change, and a knocked mount all shift the relationship during a session, and the recorded action stream continues to look plausible because the operator compensates without noticing. The result is a dataset where identical recorded actions correspond to different physical outcomes across episodes.

A proposal for a tool to detect leader and follower calibration drift in already-recorded datasets has been open since June 2026, which is the useful signal here. Detection after the fact is possible in principle, from consistency between commanded and achieved positions, but it is not something the format provides. Until that changes, drift is caught by whatever you build to catch it.

A recorded frame rate that is not the sampling rate

Timing metadata is load-bearing for any policy that consumes sequences, and it is easy to get wrong in a way nothing downstream questions. A report that dataset.fps was misleading described recording strategies writing samples at the nominal frame rate multiplied by an interpolation factor, while the metadata field continued to report the nominal figure. The issue was addressed, so this is not a live defect in current versions, and the shape of the problem is what transfers.

A dataset whose declared rate differs from its true rate is internally consistent and externally wrong. Every array has the right dimensions. Every episode loads. Any model that reasons about velocity, contact timing, or action duration learns from a time base that does not match the physical recording, and no assertion in a normal validation pass fires. Related throughput limits during high-rate observation recording can produce the same class of mismatch from a different direction, where the requested rate and the achieved rate diverge under load.

Missing annotation fields that zero the training target

The most instructive failure of the three concerns what happens when an expected field is absent. A report on SARM dense and dual training described the pipeline silently producing all-zero targets when episodes lacked subtask annotations. Training proceeded. Loss curves existed. The supervision signal was empty.

This is worth dwelling on because it inverts the usual assumption about missing data. A schema violation that raises an error costs an hour. A schema omission that resolves to a valid-looking default costs a training run plus however long it takes to suspect the data rather than the model. The general form of the risk is any optional annotation field whose absence has a numeric default, and the general mitigation is asserting presence rather than trusting shape.

Why these survive review

All three produce datasets that satisfy the checks teams normally run. Shape and dtype validation passes, because the arrays are well formed. Load tests pass, because nothing is corrupt. Replay video looks correct, because the trajectories are real recordings of real motion. Statistical summaries look reasonable, because the distributions are unremarkable.

What the three have in common is that the defect lives in the relationship between the data and its description, rather than in the data itself. Validation that inspects the recording cannot see it. Catching this class of problem requires assertions about the recording conditions, which means someone has to write down what the conditions were supposed to be. This is the same reasoning behind auditing the language annotations on an existing dataset rather than assuming a populated field is a correct one, generalized from one field to the collection protocol.

What to check while the rig is still up

Order the checks by whether the information survives teardown.

Before the session

Record a calibration snapshot as data, not as a log line, including joint offsets, camera intrinsics and extrinsics, and the achieved sample rate from a short test recording rather than the requested one. Assert that every annotation field your training code reads is present and non-default on a two-episode trial run, and confirm that removing one produces an error rather than a default.

During the session

Log commanded against achieved joint positions continuously, since their divergence is the drift signal and it is unavailable later. Re-run the calibration snapshot at intervals and after any physical interruption. Watch achieved sample rate against requested rate under real load, because throughput limits appear when the full sensor set is streaming and not during a short test.

After the session, before teardown

Re-measure calibration and store the closing snapshot alongside the opening one, which turns drift from an unanswerable question into a bounded interval. Verify that per-episode timing matches the declared rate by measuring it from timestamps rather than reading the metadata field. Confirm annotation coverage per episode as a count, not a spot-check.

The schema fields worth fixing first

Four items account for most of what cannot be recovered. Camera intrinsics have an open discussion about where they should be stored, which means teams are currently choosing individually, and an unstored intrinsic is unrecoverable once the camera moves. Per-step timing should be recorded as measured timestamps rather than derived from a declared rate. Calibration state deserves an explicit opening and closing snapshot. Subtask or phase annotation should be required rather than optional, and the request for motor primitive-level annotations points at how much semantic structure teams want that the format does not yet carry.

Everything else, including relabeling, re-chunking, and reformatting, can be done from a faithful recording. These four cannot.

Where this leaves dataset comparison

Metadata integrity also affects conclusions teams draw about policies. A reported difference in performance between two policies trained on the same real robot data is the kind of observation that gets attributed to architecture, and it is a discussion thread rather than a controlled benchmark, so it should be read as a practitioner report. It illustrates the interpretive problem well. When two policies consume timing, camera, and annotation metadata differently, identical trajectory data is not identical input, and a comparison between them is partly a comparison of how each handles the description.

That connects this to the broader argument that robot foundation models are short on specific data properties rather than on volume. Deciding what the right data means for your deployment is the first half of the work, and it pairs with questions about how many demonstrations a policy needs and what separates a manipulation dataset that works. Verifying that the data you already collected describes itself accurately is the second half, and it is the cheaper of the two. For contact-rich manipulation data the stakes are higher still, since force and timing carry more of the signal, and the same reasoning drives the data problem underneath vision-language-action models.

Instrument the collection before you scale it

If you are standing up a collection operation, the protocol and the metadata schema determine whether the data survives contact with a training run. HumanSignal Services designs collection protocols, runs the operation, and delivers datasets with the calibration, timing, and annotation state recorded as data rather than reconstructed afterward. Book a scoping conversation to walk through the protocol your task would need.