Beyond sim-to-real: the data gaps simulation can't close

Key takeaways

Simulation closes the cost and volume problem in robot data collection, and better physics engines keep narrowing the appearance and dynamics gap.

Four classes of signal stay out of reach regardless of fidelity: contact and material behavior, real sensor pathology, unmodeled human behavior, and the failure modes nobody thought to author.

Sim-to-real performance drops of 20 to 50 points are documented, and the residual is concentrated in exactly these categories rather than spread evenly.

Synthetic pipelines produce silent failures, which is why the strongest ones now add critics that themselves require human-verified ground truth to be trusted.

The practical division of labor is to simulate what you can specify and capture what you cannot, then use real data to verify the simulator rather than replace it.

Simulation is the most effective answer the robotics field has found to the cost of real-world collection. It is fast, parallel, perfectly labeled, and it does not wear out a gripper. Teams that use it well collect orders of magnitude more experience than teams that do not.

It also has a residual that does not shrink at the same rate as the rest. Sim-to-real performance drops of 20 to 50 points are documented in the literature (arXiv:2508.11117), and that residual is not evenly distributed across the task. It concentrates in specific categories of signal.

This piece is about those categories: what simulation structurally cannot produce, why fidelity improvements do not close them, and what to capture in the real world instead.

Gap one: contact and material behavior

Rigid-body dynamics simulate well. Contact does not, and most useful manipulation is contact.

The physics of a fingertip deforming against a soft object, a screw catching threads, fabric bunching, or a surface transitioning from static to kinetic friction depend on material parameters that are difficult to measure and expensive to model. Simulators approximate these with tuned constants that hold for the configuration they were tuned on.

The deeper issue is that the ground truth for validating a contact model is itself missing. Tactile and force sensing are close to absent from the robot learning corpus. In our analysis of 1,228 vision-language-action papers, one full chunk of 123 papers contained zero tactile-centric work, and the papers that do address it name the same cause: scarce aligned vision-tactile-language data (arXiv:2605.27886) and the absence of large multimodal datasets (arXiv:2507.17294).

That produces a circular problem. You cannot validate a contact simulator without real contact data, and the real contact data is what nobody has collected. Improving the simulator does not exit the loop.

Gap two: sensor pathology

Simulated cameras produce clean images. Real cameras produce rolling-shutter smear on fast motion, blown highlights when the robot turns toward a window, autofocus hunting, condensation, and dead pixels that appear after six months of vibration.

Depth sensors have their own catalogue: absorbed returns on black surfaces, specular dropouts on stainless steel, multipath artifacts in corners, and interference when two units see the same scene. These are not noise in the statistical sense, so adding Gaussian noise to a simulated depth map does not reproduce them. They are structured, scene-dependent failures, and they are part of why annotating real robotics data is a different exercise from labeling clean frames.

A policy trained only in simulation has never encountered them, and they occur disproportionately in the conditions where reliability matters. Capturing them requires running the actual sensor in the actual environment, which is a collection problem rather than a rendering problem.

Gap three: humans behaving unpredictably

Any robot operating near people inherits a distribution nobody can author. People reach into the workspace, hand over objects at odd angles, change their minds mid-motion, hesitate, and do things that make sense only with context the robot does not have.

Simulated humans follow the behavior model someone wrote, which means the policy learns to handle the behaviors the author anticipated. The cases that break deployed systems are typically the ones outside that model, and you cannot enumerate them in advance by thinking harder about the model.

This is one reason egocentric human video has become an active collection area, and why purpose-built egocentric datasets get built rather than scraped. It is also why that route is not free: human video is action-free and unstructured, so someone has to segment, ground, and align it before it trains anything, with the absence of task-aligned annotations cited as the blocker (arXiv:2606.00054).

Gap four: the failures nobody authored

A simulator contains the failure modes its author modeled. Real collection produces the ones nobody predicted, which is precisely the set worth having.

The field's habit compounds this. Researchers train on successful demonstrations and discard the rest, so even real collection often throws away its most valuable output. As VINE (arXiv:2512.03913) notes, those failures encode where and how policies are fragile, and without them minor execution errors compound into unrecoverable states (arXiv:2605.08434).

Synthetic pipelines have a specific version of this problem. They generate volume, but they lack failure coverage and produce what the literature calls silent failures: outputs that look plausible and are physically wrong. The strongest pipelines now build in model-based critics to filter their own output (arXiv:2604.09036), and those critics need human-verified ground truth to be trusted. We wrote separately about why synthetic data needs a verification layer rather than a volume argument.

Fidelity does not close these

It is worth being precise about why more realistic simulation does not solve the four categories, because the assumption that it eventually will shapes a lot of roadmaps.

Contact and materials fail on parameter identification, not rendering. Sensor pathology fails because the artifacts are properties of specific hardware in specific conditions, not of the optical model. Human behavior fails because the distribution is unauthored rather than under-resolved. Unanticipated failures fail by definition, since a simulator can only contain what someone put in it.

Each of these is a knowledge problem rather than a compute problem. Increasing fidelity improves what is already modeled. It does not tell you what you failed to model.

The world-models literature reaches a parallel conclusion from a different direction. On the Physics-IQ benchmark, the model with the best visual realism score scored last on physical understanding, and physics performance across all models topped out well below visual quality (arXiv:2501.09038). Looking right and being right came apart cleanly.

The division of labor that works

Teams getting good results tend to use simulation and real capture for different jobs rather than treating one as a cheaper substitute for the other.

Simulate what you can specify: kinematics, coarse geometry, task variation across configurations you can enumerate, and volume for pretraining priors. This is where simulation's advantages are real and the residual is small.

Capture what you cannot specify: contact and force signatures for the materials you actually handle, your specific sensors failing in your specific environment, humans behaving unscripted in the deployment space, and failures as they occur rather than as authored.

Then use the real data to audit the simulator rather than only to fine-tune the policy. Comparing simulated and real trajectories on the same task tells you where your model of the world is wrong, which is information you can act on. Fine-tuning alone hides that information inside the weights.

A practical consequence is that real collection can be small and targeted if it is aimed at the four categories. The value is in coverage of the residual, not in matching simulated volume. Tracking that coverage explicitly, with agreement measured on the judgment-heavy annotations, keeps a small real dataset honest.

Capture the residual, not the volume

If your policies work in simulation and degrade on hardware, the useful question is which of the four categories your residual sits in, and what a targeted capture would need to record. HumanSignal Services designs that collection, instruments it, and delivers a verified dataset built to your spec, including contact-instrumented and on-site multimodal capture. Book a scoping call to scope the residual you are carrying.