The data problem hiding inside embodied AI

Key takeaways

Embodied AI cannot borrow the web-scale data strategy that worked for language and vision, because the data it needs records physical interaction that was never written down anywhere.

Every trajectory a robot learns from has to be produced deliberately, on hardware, by a person, which makes the marginal cost of an example roughly constant instead of near zero.

In a HumanSignal analysis of 1,228 vision-language-action papers, an estimated 150 to 200 open by naming high-quality trajectory data as the binding constraint on progress.

The field's own diagnosis has moved from "we need more robot data" to "the data we have is poorly annotated, homogeneous, and unverified," which reframes the work as annotation and curation rather than collection volume.

Treating embodied data as an operations problem, with a collection protocol, a review workflow, and measured coverage, is what separates a dataset that transfers from one that memorizes.

Language models had the internet. Vision models had a few billion captioned images. Both fields grew up assuming that if you needed more data, it already existed and the work was retrieval and filtering.

Embodied AI inherited the architectures from those fields and almost none of the data conditions. A robot policy learns from records of physical interaction, and those records were never produced as a byproduct of human activity on the web. Someone has to make each one.

This piece is about what that difference does to a team's roadmap, and why the constraint shows up as an operations problem rather than a modeling one.

The data does not exist as a byproduct

The reason web-scale worked is that humans generated the training corpus incidentally. Nobody wrote a blog post so that a language model could learn syntax. The data was exhaust.

Physical interaction leaves almost no exhaust of that kind. A person picking up a coffee cup produces no record of the joint angles, the contact forces, the moment the grip slipped and recovered, or the intent behind the motion. Video of the same action captures what it looked like from one angle, without the action commands underneath it. Our analysis of the vision-language-action literature found the field naming this directly: a 2026 survey of datasets, benchmarks, and data engines (arXiv:2604.23001) argues that the central underexamined bottleneck is data infrastructure itself, and that progress will depend less on architecture than on the co-design of data engines and evaluation protocols.

That is a category difference, not a scale difference. You cannot crawl your way out of it.

Every example costs roughly the same as the last one

In a web-scale regime, the cost per additional example trends toward zero once the pipeline exists. In an embodied regime it does not, because each example consumes robot time, operator time, and a physical setup.

The literature is blunt about this. A sentence close to "performance is fundamentally constrained by the availability of high-quality robot trajectory data, whose collection on real robots is costly, labor-intensive, and difficult to scale" appears as the motivating claim in an estimated 150 to 200 of the 1,228 vision-language-action papers we analyzed. GigaBrain-0 (arXiv:2510.19430) argues the inefficiency of physical collection severely limits scalability. EgoVLA (arXiv:2507.12440) notes that the hardware requirement itself constrains data scale. Real2Render2Real (arXiv:2505.11917) calls teleoperation, still the prevailing paradigm, costly and constrained by manual effort and physical robot access.

Constant marginal cost changes the shape of every decision downstream. When more data is cheap, you collect broadly and filter later. When each hour of collection is expensive, what you choose to collect, and how well you annotate it, determines whether the spend produced anything useful.

More data stopped being the answer

The instinct is to treat this as a budget problem and buy more trajectories. Recent results argue against that.

A 2026 study on rethinking scaling (arXiv:2602.09722) found that pooling heterogeneous robot datasets often induces "negative transfer," where adding data from other robot embodiments makes the model worse. MPVI (arXiv:2606.00985) documents long-horizon failures that persist despite finetuning on large teleoperated datasets. A data-distillation result (arXiv:2511.16233) found a curated 5% coreset recovering 85 to 90% of full-dataset performance, which means most of the volume in current corpora is contributing very little.

The most useful finding for a team planning collection comes from SEVO (arXiv:2605.11114): deliberately varying lighting, backgrounds, and distractors during teleoperation was the single most important factor for generalization, while data collected only in-distribution produced near-zero transfer to new environments. Composition beat volume, and composition is a protocol decision made before anyone touches a robot.

The annotation layer is where the value is

If the constraint were raw quantity, the fix would be more operators. The field's self-diagnosis has moved somewhere less comfortable: the data that exists is poorly annotated, homogeneous, and unverified.

Three concrete symptoms show up across the corpus. Language annotations are repetitive and template-like, to the point where models learn to ignore them entirely; LIBERO-PRO (arXiv:2510.03827) showed outputs holding steady even when instructions were corrupted or replaced with meaningless tokens. Failure data is collected and then discarded, so models get no corrective signal, even though those failures encode where policies are fragile (arXiv:2512.03913). Whole modalities are missing, with tactile and force sensing nearly absent from a corpus that is overwhelmingly RGB, language, and proprioception.

None of those is a collection-volume problem. Each is a question about what gets recorded alongside the trajectory, and who decides what a good record looks like. That is annotation work on robotics data, and it needs people who understand the task well enough to judge it.

What treating it as operations looks like

Teams that get useful embodied datasets tend to make the same four decisions explicitly rather than by default.

Write the collection protocol before collecting

Decide in advance which conditions vary and by how much: lighting, surfaces, object instances, operator, distractors, starting states. The SEVO result says this is the highest-leverage decision available, and it is unrecoverable after the fact. You cannot add diversity to a finished dataset.

Record failures as first-class data

Sessions produce slips, collisions, and mis-grasps whether or not you keep them. Deciding up front to annotate failures with a schema, rather than deleting the episode, converts a byproduct into the corrective signal the policy needs.

Annotate intent, not only outcome

A trajectory records what happened. What the operator was trying to do, which sub-goal failed, and why a recovery worked are separate judgments that a person has to supply. This is where domain understanding stops being optional.

Measure coverage and agreement, not throughput

Hours collected is a vanity metric. What predicts transfer is coverage of the conditions you care about and consistency among the people producing the labels. Both are measurable during collection, using inter-annotator agreement on the judgment-heavy fields and explicit coverage tracking on the protocol dimensions, and both are far cheaper to fix while the rig is still set up.

Run those four together and the collection stops being a data-gathering exercise and starts being a production process with a spec, a QA step, and an acceptance test. That is the shift the field is asking for when its surveys say data infrastructure should be a first-class research problem.

The constraint is not going away

Human video corpora, synthetic engines, and cheaper teleoperation are all real answers to cost, and all three convert the problem rather than removing it. Human video is action-free and unstructured, so someone has to segment, ground, and align it before it trains anything (arXiv:2606.00054). Synthetic pipelines generate volume but produce silent failures, which is why the strongest ones now build in critics that themselves need human-verified ground truth (arXiv:2604.09036).

Every route back to scale runs through a human judgment somewhere. For teams building in this space, the practical implication is that dataset design deserves the same rigor as model design, and probably earlier in the schedule.

Bring us the collection problem, not the headcount problem

If your roadmap depends on data that has to be captured under a protocol rather than found, the protocol is the hard part: what varies, what gets annotated, how failures are recorded, and how you know the result covers what you need. HumanSignal Services designs the collection, runs the operation, and delivers a verified dataset to spec, including on-site physical AI capture. Book a scoping call to walk through what yours would take.