How do you collect dexterous manipulation data?

A rig that records contact-rich manipulation has to solve a problem no amount of camera quality fixes: the instant that carries the most information is the instant the hand is hidden. Fingers close around an object, the grip either holds or slips, and the cameras see the back of a hand. Everything a policy needs to learn about that moment has to come from somewhere other than vision, recorded at a rate fast enough to resolve it and aligned tightly enough to be trusted. If you need the definition of this data first, what this data is made of covers it. This piece is about the rig.

Key takeaways

• The hardest moment to record is the one that matters most. Cameras lose sight of the fingertips exactly at contact, which is where the task succeeds or fails.
• That makes non-visual sensing structural rather than optional. Sensing beyond RGB is one of the four properties robot foundation models are short on, and it cannot be added after collection.
• Synchronization decides whether the extra sensing is usable. A tactile stream misaligned with the trajectory teaches a false relationship, which is worse than having no tactile stream.
• Measure the achieved sampling rate rather than trusting the declared one. Recording strategies have written data at a rate different from the value the dataset metadata reported.
• Decide segmentation before recording. Missing subtask annotations have silently produced all-zero training targets rather than an error.

The problem the rig has to solve

Occlusion in most vision tasks is an inconvenience distributed across a recording. In manipulation it is concentrated precisely where the signal is. The approach phase is fully visible and largely uninformative; the contact phase determines the outcome and is the least observable part of the sequence.

That inversion is the design constraint, and it is why contact-rich work is the hardest data to capture. A rig built to maximize visual coverage will still be blind at the decisive moment. A rig built around the assumption that vision will fail there, with other modalities carrying the load, records something a policy can use.

This also changes how camera placement is chosen. The instinct is to position cameras for the clearest view of the hand, which optimizes for the phase that needs it least. A more useful arrangement accepts occlusion at contact and places cameras to capture object state and scene context instead, since those remain visible throughout and are what the non-visual channels cannot supply.

Sensing beyond the cameras

Sensing beyond RGB is one of the properties these models are short on, and unlike annotation it cannot be recovered afterward. Three channels carry most of what vision loses.

Force and torque at the wrist

A six-axis sensor between the arm and the end effector gives you the aggregate interaction between hand and world: when contact began, roughly how hard, and in what direction. It is the cheapest meaningful addition to a rig and the one most often already present. Its limitation is that it aggregates. A wrist sensor cannot distinguish a firm grip from two fingers pressing against each other, because both produce similar readings at the wrist.

Tactile and fingertip pressure

Sensing at the contact surfaces resolves what the wrist aggregates: which fingers are loaded, whether pressure is distributed or concentrated, and whether an object is slipping. This is the channel that most directly substitutes for the occluded view, and the one with the widest variation in what hardware provides. Record the raw signal rather than a derived contact flag, because a threshold chosen during collection cannot be revisited and a raw stream can always be thresholded later.

Proprioception and its limits

Joint positions and commanded torques come free with the robot and are the easiest channel to over-trust. Commanded position is not achieved position, and the difference is exactly the interaction you are trying to capture. Record both streams. Their divergence is a signal about contact, and it is also how calibration drift becomes detectable after the fact.

Synchronization is the whole game

Multiple sensor streams at different rates on different clocks create an alignment problem, and alignment failure is worse than absence. A tactile reading attributed to the wrong frame teaches that a grip tightened before the hand arrived. The model learns the relationship as recorded, and nothing downstream flags it, because every stream is individually well-formed.

Two disciplines follow. Timestamp every stream against one clock at the source rather than reconstructing alignment from nominal rates. Then measure the achieved rate rather than trusting the declared one: a report on LeRobot described `dataset.fps` misreporting the true rate because recording strategies applied an interpolation multiplier, which was addressed, and the shape of that problem recurs anywhere a rate is declared rather than measured. Related failures of timing and calibration that survive review are the same class of defect: internally consistent files describing the recording incorrectly.

Specifying a tolerance in milliseconds is task-dependent and not something to copy from another project. The workable rule is that alignment error must be small relative to the duration of the contact events you care about, which for fast slip events is demanding and for slow insertions is not.

Designing the capture protocol

Decide segmentation before you record

Settle the task taxonomy and subtask boundaries before the first session, and have operators mark them during capture. Retrofitting boundaries means someone inferring intent from video they did not perform. The schema question is not cosmetic: a report on SARM training described a pipeline silently producing all-zero targets when episodes lacked subtask annotations, so training ran against an empty signal and reported nothing wrong.

Record contact events as first-class data

Treat contact onset, release, regrasp, and slip as labeled events with timestamps rather than as something to be inferred later from force traces. Operators can mark them, and threshold detectors can propose them for review. Either way they belong in the schema, since what separates a manipulation dataset that works is usually the presence of this structure rather than raw volume.

Capture failure and recovery deliberately

Slips, drops, and regrasps are the behaviours a deployed policy most needs and the ones a tidy collection protocol quietly excludes. Script sessions that induce them, keep the episodes, and label the cause. A dataset of clean successes teaches a policy nothing about recovering from the failure mode it will meet first.

What to verify while the rig is still up

Before the first production session, assert that every field your training pipeline reads is present and non-default on a two-episode trial, and confirm that deliberately removing one raises an error rather than resolving to a default. Record camera intrinsics and a calibration snapshot as data rather than as a log line, since where intrinsics should live is still an open schema question in the tooling and an unstored intrinsic is unrecoverable once a camera moves. Measure the achieved sample rate for every stream under full sensor load rather than in a short test, because throughput limits appear when everything is streaming at once.

Take a closing calibration snapshot at the end of each session too. Two snapshots turn drift from an unanswerable question into a bounded interval.

One more check is worth building into the routine: replay a recorded episode against the raw streams and confirm that the contact events line up with the force traces and the video. This catches alignment errors that no per-stream validation finds, because each stream is individually well-formed and only their relationship is wrong. Running it once per session costs minutes and is the only practical way to notice a synchronization fault before it is baked into a month of recordings.

Where human video fits

Human capture scales in ways a robot rig does not, and EgoDex recorded 829 hours across 338,000 episodes and 194 tasks with paired 3D hand and finger tracking, entirely tabletop. It is a strong source of hand motion and task structure at volume, and it carries no force or tactile signal, which is the channel this whole piece is about.

The practical arrangement is complementary rather than substitutive: human video for breadth of task and motion, instrumented robot collection for the contact physics, with attention to whether the capture is diverse enough to generalize on both sides.

Design the rig before the first session

Sensor selection, synchronization, and schema are decided once and constrain everything afterward, which makes them worth getting right before a rig goes up. HumanSignal Services designs contact-rich collection protocols, runs the operation, and delivers episodes with force, tactile, timing, and contact events recorded as structured data. Book a scoping conversation to work through what your task needs from a rig.