Why contact-rich data is so hard to collect, and why it matters

Key takeaways

Contact is where most manipulation tasks actually succeed or fail, and it is the signal least likely to be present in a robot dataset.

In a HumanSignal analysis of 1,228 vision-language-action papers, one 123-paper sample contained zero tactile-centric work, and audio appeared in three efforts across the whole corpus.

The barrier is instrumentation and time alignment rather than annotation effort: tactile and force streams need hardware most rigs lack and synchronization most pipelines do not handle.

Standard teleoperation systematically under-represents contact, because the operator cannot feel what the robot feels and compensates visually.

Contact data is expensive per episode and small in useful volume, which makes protocol design and verification more important than throughput.

Watch a person insert a plug, seat a bearing, or peel a label and the visually interesting part is brief. The part that determines success is what happens in the last few millimeters, where the object resists, slips, catches, or seats.

Robot datasets are largely blind to that moment, which is one of the recurring themes in our physical AI research. They record RGB video, language, and proprioception, which capture where the arm was and what it looked like, not what it felt. For tasks where vision cannot tell you whether contact succeeded, this is the difference between a policy that works and one that hunts.

This piece covers why that data is missing, what makes it hard to capture, and what a collection designed to produce it has to do differently.

The data is genuinely absent

This is not a case of a modality being under-used. It is close to unrepresented.

In our analysis of the vision-language-action literature, the corpus is overwhelmingly RGB plus language plus proprioception. Tactile and force sensing are nearly absent, to the point where one full chunk of 123 papers contained zero tactile-centric work. Audio is thinner still, appearing in three efforts across 1,228 papers.

The papers that do address contact name the cause consistently: scarce aligned vision-tactile-language data (arXiv:2605.27886) and the absence of large multimodal datasets (arXiv:2507.17294). The few datasets that exist, including HapTile, ForceVLA, and OmniVTLA, are treated as flagship contributions in their papers, which is a reliable signal that the data does not otherwise exist.

Scarcity of this kind is self-reinforcing. Without datasets, few models use the modality; without models that use it, few teams instrument for it.

Why capturing it is hard

Four barriers stack, and they are mechanical rather than editorial.

The sensors are not standard equipment

A camera is a commodity. Tactile sensing is not. Optical tactile sensors, capacitive arrays, and joint-torque estimation each measure something different, with different spatial resolution, temporal response, and failure behavior. Fingertip sensors wear against abrasive surfaces and need recalibration. Force-torque units at the wrist measure the sum of what the whole hand experienced, which is a coarser signal than per-finger contact.

Choosing among them is a task-specific engineering decision made before collection, and it constrains everything downstream. The same holds for audio, where a click or a scrape marks a state change and labeling it requires the audio and video streams side by side. A dataset recorded with a wrist force-torque sensor cannot be retroactively upgraded to per-fingertip resolution.

Alignment is unforgiving

Contact events are short. A slip or a catch can resolve in tens of milliseconds, which means a tactile stream misaligned with video by even a modest offset attributes the event to the wrong frame.

Vision at 30 frames per second, tactile at several hundred hertz, and joint states at another rate all have to be synchronized to a common clock with known latency per channel. Doing this properly is a hardware and software problem at collection time. Doing it badly produces a dataset that looks complete and teaches the wrong associations, which is worse than not collecting the modality at all. Tooling that can display and label several synchronized streams together is a prerequisite for anyone reviewing this data rather than trusting it blind.

Teleoperation hides contact from the operator

This one is under-appreciated and structural. In standard teleoperation the operator sees a video feed and has no force feedback, so they cannot feel resistance, slip, or a part seating.

Operators compensate visually. They approach slowly, use exaggerated clearances, retry on visual cues, and avoid the fast confident motions a person uses when they can feel what is happening. The result is that the demonstrations are systematically biased away from the contact regimes the policy needs to learn. Teleoperation is already the prevailing paradigm and already costly, constrained by manual effort and physical robot access (arXiv:2505.11917), and on contact tasks it also distorts what gets recorded.

Bilateral or force-feedback teleoperation addresses this and raises rig cost and complexity again.

Ground truth is a judgment

For a bounding box, two annotators agree or they do not. For contact, the useful labels are interpretive: did the part seat, did the grip slip and recover, was the applied force appropriate, at what moment did the state change.

Someone has to define those categories, and someone with task understanding has to apply them. Since the labels involve judgment, they need agreement measurement to be trustworthy, and low agreement usually means the schema is under-specified rather than that the annotators are careless.

Why it matters more than the volume suggests

Contact data is expensive per episode, and the useful quantity is smaller than teams expect, for a reason that makes the investment more attractive rather than less.

Contact is where the failures concentrate. A policy that misjudges approach trajectory usually recovers; a policy that misjudges contact drops the part, jams the assembly, or applies force in a way that damages something. The failure modes are both more common and more consequential, and they are exactly the episodes the standard pipeline deletes, since researchers train on successes and discard the rest even though those failures encode where policies are fragile (arXiv:2512.03913).

Simulation does not close the gap either. Contact physics depends on material parameters that are hard to identify without real measurements, which creates a loop: validating a contact simulator needs real contact data, and real contact data is what is missing. Documented sim-to-real drops of 20 to 50 points (arXiv:2508.11117) concentrate in exactly these regimes.

What a contact collection has to do differently

If you are specifying one, five things change relative to a standard manipulation collection.

Choose sensing to match the failure mode you care about. Per-fingertip resolution for slip and in-hand manipulation, wrist force-torque for insertion and seating, and audio where a click or scrape marks a state change.

Synchronize to a common clock and record per-channel latency as metadata. Verify alignment on a known event before running the full campaign, not after.

Use force-feedback teleoperation where budget allows, or accept and document the visual-compensation bias if it does not.

Keep failures and annotate them with a schema covering what went wrong, at which moment, and whether recovery succeeded.

Define contact-state categories up front and measure agreement on them during collection, while the setup still exists and the schema can be revised.

Volume targets matter less here than coverage of material and geometry combinations. Twenty carefully instrumented episodes across ten material pairings will usually teach more than two hundred episodes on one.

Instrument the collection before you scale it

Contact data is the clearest case of a dataset that cannot be assembled from existing sources, cannot be simulated without real measurements to calibrate against, and cannot be repaired after the rig comes down. HumanSignal Services designs the protocol, instruments and runs the collection, and delivers a verified dataset built to your spec. Book a scoping call to work through the sensing and alignment requirements for your task.