Why the last 5% of cases is a different data problem entirely

Key takeaways

The first 95% of cases and the last 5% are solved by different methods, because the tail cannot be reached by collecting more of the same distribution.

Rarity is a property of the sampling process, so scaling collection scales the head and leaves the tail roughly as sparse as it was.

In driving data, collisions and near-misses are described as exceedingly sparse in real logs, which leaves models weakest on the interactions that carry the most risk.

Tail cases have to be constructed, sought deliberately, or salvaged from failures the standard pipeline discards, and each route has a different cost structure.

Aggregate accuracy hides tail performance by construction, so tail work needs its own evaluation set and its own acceptance criteria.

Most machine learning projects follow a recognizable curve. Progress is fast early, the model reaches a level that demos well, and then improvement slows sharply while the remaining failures stay stubbornly present.

The usual response is to collect more data, which worked every previous time. In the tail it mostly does not, and the reason is worth understanding precisely, because it changes what you should spend money on.

Rarity is a property of your sampling process

If an event occurs in one out of every two thousand instances in the world, and you collect by sampling the world, then doubling your collection budget gets you twice as many instances of that event and twice as many of everything else. The proportion is unchanged.

That is the whole problem in one sentence. Scaling collection scales the head. The tail arrives at the same rate it always did, which means the ratio driving your model's attention during training stays where it was.

Waiting is not a strategy either, because the tail is not a fixed set. New object types appear, environments change, and the population of edge cases shifts underneath a deployed system.

What the tail looks like in practice

The clearest documented example comes from driving. A survey of the area describes collisions and near-misses as exceedingly sparse in real logs (arXiv:2501.11260), which produces a specific and uncomfortable result: models trained on logged driving data are weakest on exactly the interactions that carry the most risk.

Robotics shows the same shape. Long-horizon failures persist despite finetuning on large teleoperated datasets (arXiv:2606.00985), and minor execution errors compound into unrecoverable out-of-distribution states because the policy has never observed a recovery (arXiv:2605.08434). Both are tail phenomena. The common case was covered; the compounding sequence was not.

The pattern generalizes past physical systems. In document processing it is the unusual layout, in content review the ambiguous case that policy did not anticipate, in speech the accent and acoustic condition combination that never appeared in the corpus. The domain changes and the structure does not. It shows up in agent behavior too, where evaluating multimodal agents means covering the trajectories that rarely occur rather than the common path.

Aggregate metrics conceal it

Part of why teams underinvest here is that the standard measurements are designed to hide tail performance.

If the tail is 5% of your traffic, a model that fails on all of it still reports 95% accuracy. The metric is dominated by the head, and improvements in the head continue to move it while the failures that generate incidents stay untouched.

The evaluation literature in robotics makes a sharper version of this point. MetaFine (arXiv:2605.19986) argues that collapsing capability into binary success rates inflates reported performance by up to 70%, and real-world evaluations typically rest on 25 or fewer rollouts without confidence intervals (arXiv:2605.29710), which is too few to resolve tail behavior at all. LIBERO-Plus found success rates dropping from around 95% to below 30% under modest perturbations (arXiv:2510.13626), which is the tail asserting itself the moment conditions move.

The implication is that tail work needs a separate evaluation set with its own acceptance criteria. Measuring it inside the aggregate guarantees it stays invisible, and a benchmark that evolves as the model improves is the version of this that keeps working past the first release.

Three routes into the tail

Since sampling more will not get you there, the options are to construct the cases, seek them deliberately, or recover the ones you were already discarding.

Construct them

Author the cases directly, in simulation or through staged real-world capture. This gives full control over coverage and lets you enumerate combinations that would take years to encounter.

The limit is that you can only construct what you thought of, and the tail is partly defined by containing what nobody anticipated. Constructed tail data also inherits the sim-to-real residual, with documented drops of 20 to 50 points when moving to hardware (arXiv:2508.11117). Useful for the known unknowns, silent on the rest.

Seek them deliberately

Run collection targeted at where the tail concentrates rather than sampling uniformly: the unusual site, the difficult lighting, the demographic or acoustic condition the corpus is thin on, the operating regime near the system's limits.

This is more expensive per hour than general collection and much cheaper per useful example, which is the trade that makes it worth doing. It requires knowing where your tail is, which usually comes from production failures rather than from planning.

Recover what you discard

The cheapest source of tail data is frequently already in your pipeline and being deleted.

Collection sessions produce failures continuously, and the standard practice is to train on successes and throw the rest away. Those failures encode where policies are fragile (arXiv:2512.03913), and the datasets now built specifically to supply them, such as RoboFAC with 9,440 erroneous trajectories (arXiv:2505.12224) and ViFailback with 58,000 failure-diagnosis pairs (arXiv:2512.02787), exist because the norm discards this signal.

The same logic applies to production. Low-confidence predictions, human overrides, escalations, and reversals are tail cases arriving with a free label attached. Routing them into a review queue costs a workflow rather than a collection campaign.

Tail annotation is a different job

One more asymmetry decides whether tail work succeeds: the labeling is harder, and it is harder in a way that changes who can do it.

Head cases are usually unambiguous, which is why throughput-oriented annotation works on them. Tail cases are ambiguous by construction. If they were obvious they would be in the head. The annotator has to decide what an unusual situation even is before labeling it, and that decision requires understanding the domain rather than following a rubric.

Two consequences follow. Agreement between annotators drops on tail data, and low agreement should be read as a signal about the schema rather than about the annotators. Measuring it with a proper agreement statistic tells you which categories are genuinely underspecified. And throughput falls, so a tail queue budgeted at head rates will be either rushed or abandoned. Separating the queues, with review workflows sized for the harder work, is usually the practical fix.

Budget the tail as its own project

Treating the last 5% as a continuation of the first 95% is what produces the plateau. It has a different acquisition strategy, a different annotation profile, and a different evaluation set, which makes it a project rather than a phase.

Teams that plan it that way tend to find the tail cheaper than expected, because targeted collection and recovered failures are both efficient once someone is looking for them. Teams that do not tend to spend the same money on more head data and wonder why the incident rate holds steady.

Scope the tail before it scopes you

If your model demos well and fails in production, the cases causing incidents are unlikely to arrive through more of the same collection. HumanSignal Services designs targeted collection for exactly those conditions, runs it, and delivers a verified dataset built to your spec. Book a scoping call to work through where your tail actually sits.