How do you build a diverse egocentric dataset?

Egocentric capture has become cheap enough that hours are no longer the constraint. A head-mounted rig and a willing participant will produce more first-person video than a team can annotate, and the resulting dataset can still fail to teach a model anything about the conditions it will meet in deployment. The reason is that scale and diversity are bought separately, and capture that scales easily tends to scale along one axis while staying flat on the rest. This piece names the axes that vary independently, shows what the published datasets cover, and sets out how to specify diversity before a camera is switched on.

Key takeaways

• Scale and diversity are separate purchases. EgoDex runs to 829 hours and 338,000 episodes across 194 tasks, and consists entirely of tabletop manipulation.
• Breadth looks different. Ego4D gathered 3,670 hours from 923 participants across 74 locations in 9 countries, with participants aged 18 to over 80, spanning household, outdoor, workplace, and leisure settings.
• Five axes vary independently: participants, environment and geography, task structure, capture hardware and viewpoint, and conditions such as lighting, clutter, and failure. A dataset can be enormous along one and flat along the rest.
• Diversity has to be a quota written into the protocol, not an outcome of opportunistic capture, because convenience sampling concentrates on the easiest conditions.
• Record the axis values per episode. Calibration, intrinsics, and capture conditions cannot be reconstructed once collection ends, so an unrecorded axis is one you cannot analyse or rebalance later.

Scale and diversity are separate purchases

Two of the largest egocentric datasets illustrate the split cleanly, and neither is deficient; they were built for different purposes.

EgoDex, collected with Apple Vision Pro, contains 829 hours of 1080p video at 30 Hz, 338,000 episodes across 194 tasks, roughly 90 million frames, with paired 3D hand and finger tracking captured at recording time. By the authors' own description it consists entirely of active tabletop manipulation, ranging from tying shoelaces to folding laundry. That is enormous depth within one capture condition, and for learning dexterous hand motion it is exactly the right shape.

Ego4D made the opposite trade. It gathered 3,670 hours from 923 unique participants across 74 locations in 9 countries, with participants ranging in age from 18 to over 80 and spanning scenarios including household, outdoor, workplace, and leisure activity. Fewer hours per condition, far more conditions.

A model trained only on the first will encounter deployment conditions it has never seen. A model trained only on the second may never see any single task often enough to learn it precisely. Neither dataset is wrong, and a team that treats "hours collected" as the headline number will not notice which of the two it has built.

Five axes that vary independently

Diversity is not one quantity. Treating it as a single dial is what produces datasets that are broad in a way nobody needed.

Participants

How many distinct people, and how varied in body size, handedness, height, age, and skill level. Hand geometry and reach envelope differ enough between participants to change what a manipulation policy learns. Ego4D's 923 participants across an 18 to 80-plus age range is the high-water mark here; many task-focused sets use fewer than ten.

Environments and geography

Distinct physical spaces, not distinct recordings in one space. A dataset shot in four rooms of one building has one lighting design, one furniture vocabulary, and one set of surface materials. Geography adds another layer, because household object inventories and room layouts differ substantially between regions.

Task and activity structure

Whether the dataset covers one activity class deeply or many shallowly, and whether tasks include multi-step sequences with dependencies rather than isolated actions. This axis is where EgoDex and Ego4D separate most sharply.

Capture hardware and viewpoint

Headset versus chest mount versus glasses, field of view, frame rate, resolution, and whether depth or hand tracking is captured natively. A model trained on one rig's intrinsics and mounting height learns that viewpoint along with the task, which becomes visible only when the hardware changes.

Conditions: lighting, clutter, failure

Time of day, artificial versus natural light, occlusion, background motion, and whether the participant ever fails or corrects. Deliberately capturing failure is the least common practice and among the most useful, since deployment is where things go wrong.

These five are worth tracking separately because they trade against each other rather than moving together. Recruiting more participants usually means fewer hours each. Adding buildings raises travel and setup cost per hour. Widening the task set reduces episodes per task. A protocol that asks for all five at once without naming which dominates will quietly optimize for whichever is cheapest to satisfy, which is almost always more hours in the sites already booked.

What the published datasets cover, and what they do not

Set the numbers side by side and the pattern is consistent: datasets are strong on the axes their builders needed and thin on the rest. This matters when you are evaluating whether to reuse a public dataset rather than commission your own, because the headline hours figure tells you almost nothing about whether the coverage you need is present. EgoDex is exceptional on task count and hand-tracking fidelity within one environment class. Ego4D is exceptional on participants and geography with less per-task density.

It is worth a comparison from outside egocentric capture entirely. DROID, which is teleoperated robot data rather than first-person human video, was built with scene diversity as an explicit design goal: 564 scenes across 52 buildings, covering 86 tasks, collected by 50 collectors over twelve months. The lesson transfers even though the modality does not. Scene count rose because it was a stated target of the protocol, not because collection happened to spread out.

This is the concrete form of the argument that robot foundation models are short on condition diversity as a named missing property rather than on volume, and it is the same reason scraped video does not substitute for designed capture. Found footage carries whatever distribution it happened to have.

Specifying diversity before you collect

Write the axis targets into the protocol

State a number per axis before collection: distinct participants, distinct buildings, task classes, rig configurations, and lighting conditions. A protocol that says "varied household settings" produces whatever the recruiters found convenient. A protocol that says "no more than 15% of hours from any single building" produces a measurable property.

Sample against a quota, not opportunistically

Track coverage per axis during collection and route new sessions toward the thin cells. Opportunistic capture concentrates on whoever is available and whichever spaces are easiest to book, which correlates with easy lighting, tidy rooms, and cooperative objects. Quota sampling costs more per hour and is the only way to prevent the correlation.

Record the axis values as metadata

Every episode should carry participant identifier, environment identifier, task class, rig configuration, and condition flags. Without these fields you cannot measure coverage, rebalance a dataset, or hold out a condition for evaluation.

The held-out condition is the part teams skip and later want. Reserving one building, one participant group, or one lighting regime entirely from training is what lets you measure whether a policy generalizes across that axis rather than within it. That split has to be decided before collection, because reserving a condition only works if you captured enough of it to evaluate on and never trained against it. This is the same discipline behind what cannot be reconstructed once the rig comes down, applied to diversity rather than to calibration.

What cannot be fixed afterward

Annotation can be added later. Labels can be corrected, language descriptions written, and formats converted at any point from a faithful recording.

Coverage cannot. If your dataset contains four buildings, no downstream process produces a fifth. Camera intrinsics, mounting geometry, and lighting conditions are properties of the moment of capture, and an unrecorded participant identifier cannot be reverse-engineered from video. Skew discovered during evaluation can only be fixed by collecting again, which is why the axis targets belong in the protocol rather than in the analysis plan. Anyone weighing where to source this data is really choosing which coverage they will be able to obtain, and that decision is upstream of everything else, as the data problem inside embodied systems usually turns out to be.

Design the capture around the axes you name

Coverage is a protocol decision, and protocol decisions are cheapest before the first session. HumanSignal Services designs egocentric collection against stated per-axis quotas, runs the operation across participants and sites, and delivers episodes with condition metadata recorded rather than inferred. See how household capture is run to a quota, or book a scoping conversation to work through the axes your deployment implies.