What is dexterous manipulation data?

A humanoid arm passes every simulation benchmark for unscrewing a bottle cap. It runs perfectly in hundreds of thousands of virtual trials. Then someone puts it in a kitchen. The arm grips the cap, applies torque, and strips it. The hardware is fine. The policy had been trained on data missing contact force and fingertip pressure readings, with trajectory labels at a temporal resolution that didn't match the task. The sensors were present during collection. They just weren't structured to teach what successful contact feels like.

TL;DR

Dexterous data combines movement, touch, and visual sensors into a single stream.

It is harder to collect than standard data because cameras lose sight of fingertips at the moment of contact.

Large-scale human video predicts how well a robot will perform complex manipulation tasks.

Training happens in two stages: broad human demonstrations first, then specific robot tuning.

Data only becomes useful for training once it is properly labeled and segmented.

Defining dexterous manipulation data

Dexterous manipulation data differs from standard robotics sensor data in a specific way: the word "dexterous" changes what must be captured and what must be labeled.

A basic pick-and-place robot generates joint angles and camera frames. A dexterous manipulator adds high-frequency contact force readings, tactile sensor images, and finger joint states across 16 to 24 degrees of freedom. The temporal relationships between all of them are part of the data. Learning-based methods need large, high-quality datasets because the action space of a multi-fingered hand is too complex for manual control logic (dexterous manipulation survey).

Grasping a wine glass requires the same visual affordance as grasping a plastic cup. The difference lives in how the robot adjusts force at the fingertips every millisecond, movements cameras can't see. Tactile sensors with dense fingertip arrays are now standard in research systems. They provide the contact signals that distinguish a stable grasp from one about to fail (GR-Dexter, arXiv 2512.24210).

Why capturing it is harder than other robotics data

Occlusion at the moment of contact

Contact happens where fingers meet object, exactly the region that fingers block from every external camera. The instant a grasp engages, the visual stream loses the information it most needs. Overhead cameras show the top of a closed hand. Wrist cameras show the palm. DexCap solves this by replacing vision-based tracking with SLAM-based motion capture, tracking 16-dimensional joint positions on LEAP robot hands regardless of camera coverage.

Multi-modal synchronization

Tactile sensors stream at 100 Hz. RGB cameras stream at 30 Hz. Force-torque sensors at the wrist stream at 1 kHz. To synchronize all three, timestamps must align within a millisecond so every modality refers to the same physical instant. Misaligned streams produce training examples where the label "successful grip" attaches to the force reading 50 milliseconds before the grip engaged. The policy trains on noise. GR-Dexter's four-source data mixture exists partly for this reason.

The DoF rendering problem

Standard annotation tools were built for 2D bounding boxes and class labels. A dexterous hand with 24 degrees of freedom produces a state vector that a timeline slider and a label dropdown cannot meaningfully represent. Without a purpose-built interface that renders the full kinematic state, an annotator reviewing a failed grasp replay cannot see where the ring finger's proximal joint deviated.

The simulation gap

Simulation appears to solve all three problems. Teams can generate millions of synthetic grasps, skip the hardware, and label procedurally. But simulated friction and tactile feedback don't transfer to real fingertip sensors and real surfaces. Synthetic data works for pretraining perception: it teaches a model what a glass looks like from many angles. It can't yet replace real contact data for fine-tuning, where grasps need to hold under actual force variation.

The three collection approaches and their tradeoffs

Teleoperation

A human operator controls the robot in real time using VR controllers or data gloves, while the robot's sensors record everything. Teleoperation produces data with full sensor fidelity, but throughput is limited by operator cognitive load and control interface latency. Average data collection cost per hour fell 60 percent between 2024 and 2026, driven by improved tooling and commodity hardware (State of Robotics 2026). The tradeoff: grasp configurations a policy most needs to generalize take disproportionately long to elicit from operators who default to familiar patterns.

Motion capture and egocentric video

Mocap systems and head-mounted cameras record humans performing manipulation tasks without a robot present. DexCap demonstrated 3x faster data collection throughput compared to traditional teleoperation and showed that autonomous policies can be learned from as little as 30 minutes of human mocap data. The bottleneck is retargeting: human hand kinematics don't map directly to robot actuator kinematics. Post-processing to convert human joint angles into robot-executable trajectories introduces its own error. The data also lacks the contact forces a robot's sensors would capture.

Synthetic generation

Procedurally generated grasps in simulation cover rare configurations efficiently and cost almost nothing per sample. Imitation learning surpassed reinforcement learning as the primary training method for manipulation tasks in 2026, driven by human demonstration data (State of Robotics 2026). Synthetic data earns its place in pretraining. For fine-tuning across real surfaces and object variations, real contact data is still required.

How multi-source data mixtures feed modern training pipelines

Stage one: pretraining on human video at scale

A Vision-Language-Action (VLA) model trained on 20,854 hours of egocentric video followed a log-linear scaling law: as data volume grew, validation loss fell, and that validation loss correlated strongly with real-robot performance on high-DoF dexterous tasks (EgoScale). The dataset was 20 times larger than prior efforts.

VLA model adoption tripled between 2024 and 2026 and now appears in 40 percent of all new robot deployments (State of Robotics 2026). Pretraining on human manipulation video captures the same physical intuitions as robot demonstrations at far lower cost.

Stage two: mid-training for robot-specific alignment

The scaling law holds for pretraining but doesn't eliminate the need for robot-specific data. A model pretrained on human video hasn't learned what a specific robot's fingertip sensors read during a successful pinch grasp, or how a backdrivable joint responds differently from a rigid one.

EgoScale's two-stage transfer recipe addresses this: large-scale human pretraining followed by lightweight human-robot mid-training. The mid-training stage uses a small amount of teleoperation data to adapt the model to a specific robot's sensors and controls. It requires far less data than pretraining, but the structure differs: real contact readings, correctly synchronized, labeled at the frequency the policy trains on.

No single collection method covers both stages. Egocentric human video scales cheaply but has no tactile channel and no robot kinematics. Teleoperation carries full robot sensor fidelity, but throughput limits how much a team can generate. GR-Dexter takes this approach with four sources: vision-language, cross-embodiment, human trajectories, and robot trajectories. Pretraining needs scale; mid-training needs tactile precision.

What annotation adds that sensors alone can't

A stream of tactile readings, joint angles, and video frames tells a policy nothing on its own. Someone must define which episodes succeeded, which force signatures indicate incipient slip, and which 30-second segment contains the technique worth imitating.

Episode segmentation divides a continuous recording into task-relevant chunks with semantic boundaries ("approach," "contact," "hold," "release"), not raw timestamps. Quality scoring filters out episodes where the operator struggled or the hardware slipped. Subtask labeling marks which part of an episode contains which grasp strategy, so the model generalizes across grip types.

Work performed at the wrong granularity corrupts the training signal. An episode labeled "successful bottle cap removal" that includes 20 seconds of failed attempts is noisy at best. At worst, it contradicts what the policy should learn. Labels at lower temporal resolution than the policy trains on produce the same gap the bottle cap arm hit: the annotation exists, but it doesn't resolve finely enough to teach contact.

Useful annotation requires an interface that unifies multi-camera video, subtask timelines, and quality scoring. Without that unified view, an annotator can't judge the nuances of a 24-DoF hand. HumanSignal's robotics episode review interface brings those elements together in one session. Following its acquisition of Erud AI, HumanSignal extended this into embodied AI data services covering synchronized motion and sensor data from physical environments. Generic bounding-box tools don't render kinematic states and don't support multi-modal timeline review.

Training-ready signal, not more sensor data

The simulation-trained arm failed the bottle cap because the policy had never encountered what slip feels like at the fingertip. It didn't know which joint-level state signaled a borderline grasp, or which 30-second segment held the technique that worked. The data existed. It wasn't structured to teach anything.

Dexterous manipulation data is not a quantity problem. Teams can collect thousands of hours of egocentric video and still train a policy that fails on real surfaces. Without synchronized contact readings in the mid-training data, pretraining alone doesn't close the gap. They can build a perfect teleoperation rig and still get noise in the training set if episodes aren't segmented at the right granularity. What turns raw streams into training-ready signal is the annotation layer: human decisions about which moments mattered, at what frequency, and across which modalities.