Why some training data needs subject-matter experts, not annotators

Key takeaways

On most annotation tasks the label is an observation. On some, the label is a judgment that depends on experience no guideline can transfer.

Three tests separate the two: whether a careful non-expert could reach the answer from written instructions, whether disagreement between experts is informative, and whether the consequences of a wrong label are asymmetric.

Agreement statistics measure consistency, not correctness, so a pool of non-experts can agree confidently and be wrong in the same direction.

Expert annotation needs a different workflow: rubric decomposition, captured rationale, preserved disagreement, and adjudication by a second expert rather than by vote.

Expert time is the constrained resource, so the design goal is to spend it only on the judgments that require it.

Most annotation is recording an observation. Is there a pedestrian in this frame, does this sentence express a complaint, is this document an invoice. The answer exists in the data, and a careful person who has read the instructions can find it.

A smaller set of tasks works differently. The answer is not in the data waiting to be recorded. It is produced by someone applying knowledge to the data, and two qualified people can look at the same item and reach different conclusions for defensible reasons.

Confusing the second kind for the first is expensive, and the failure is quiet, because a non-expert pool will still return labels on schedule.

Three tests that separate the two

Could a careful non-expert reach the answer from instructions?

This is the practical version of the question. Write the instructions you would give, hand them to someone smart who does not know the domain, and see whether they can do the task.

For bounding boxes, sentiment on clear examples, or document type classification, they can. For whether a radiology finding is clinically significant, whether a contract clause is unusual for its industry, whether a code change introduces a subtle regression, or whether a manufacturing defect will matter in service, they cannot, and no amount of additional guideline text closes it. The instruction would have to contain the expertise.

Is disagreement between qualified people informative?

On observation tasks, disagreement means someone made a mistake, and resolving it by vote is correct.

On judgment tasks, two experts can disagree because they weight different considerations, and that disagreement is information about the item rather than error in the process. Collapsing it to a majority destroys the distinction between a contested case and a clear one. Where you see stable, defensible disagreement among qualified people, you are on a judgment task, and the structure of that agreement is worth preserving rather than flattening.

Are the consequences asymmetric?

Where a wrong label in one direction is much worse than in the other, the expected cost of a cheap wrong answer rises sharply. Medical, legal, safety, and compliance work usually sit here, and the economics that make crowd annotation attractive stop applying.

Agreement will not save you

The most common way teams get this wrong is trusting an agreement number.

Agreement measures whether annotators are consistent with each other. It says nothing about whether they are consistent with the truth. A pool of non-experts working on a domain task will often share the same misunderstanding, because they are reasoning from the same incomplete instructions, so they agree with each other and are wrong together.

A chance-corrected agreement statistic is still worth computing, and it still cannot detect this. The only reliable check is a set of held-out items with expert-verified answers, built before the campaign starts, scored against the pool's output. If agreement is high and gold accuracy is low, the task needed expertise.

The gap is visible in the literature

The pattern is not confined to any one domain. In our analysis of the human preference data literature, a field built entirely on recorded judgment, the identity of the person judging was recorded in only 6 of 24 papers read in full. Preference was recorded as a judgment with no judge attached, which makes it impossible to know whether the label reflects informed assessment or an arbitrary click.

The same analysis found that not one of those 24 papers checked whether disagreement between annotators represented stable individual judgment rather than measurement noise. The test is inexpensive: show the same item to the same person weeks apart and see whether they are consistent, or hold out a rater and try to predict their calls. The field's most-used concept turned out to be its least-tested one.

For anyone buying or building judgment-heavy data, those two findings translate directly into requirements. Record who judged, and test whether their judgments are stable.

What expert annotation requires

Running a pool of experts like a crowd pipeline wastes the expertise. Four things change.

Decompose the judgment into a rubric

Asking an expert for an overall rating produces a number that cannot be audited or debugged. Breaking the judgment into named dimensions, each scored separately, produces something you can inspect, disagree with productively, and train on with more signal.

Rubric design is itself expert work, and it is worth doing before annotation rather than during. It also makes the task teachable, which is what lets you extend the pool later.

Capture the rationale

A label without a reason is difficult to use and impossible to check. A short structured rationale, captured at judgment time, is what makes the dataset auditable and what lets a second expert evaluate the first one's reasoning rather than only the conclusion.

This is also the piece most often skipped for throughput reasons, and the one teams most often wish they had. It is where domain experts turn observed behavior into something that improves a model rather than just scoring it.

Preserve the disagreement

Store per-expert judgments rather than only the aggregate, and record the aggregation rule you use downstream. On judgment tasks the distribution of expert opinion is closer to ground truth than any single collapsed value, and it is unrecoverable once discarded.

Adjudicate rather than vote

When experts disagree, route the item to a more senior reviewer who sees both judgments and the rationales, and who records a decision and a reason. That is more expensive per item than a majority vote and it produces a defensible label plus a precedent that sharpens the rubric for everyone.

Spend expert time only where it is needed

Experts are the constrained resource in any of this, so the design problem is minimizing the number of judgments that require one.

Most work in a judgment-heavy project is not judgment. Data preparation, segmentation, obvious cases, and routine checks can be done by a trained annotation team or automated, leaving the expert to handle the items that genuinely need them.

A workable division looks like: automated or model-assisted pre-processing for the routine layer, trained annotators for structured work that follows a tested rubric, expert review on a sampled or triaged subset, and senior adjudication on disagreement. That structure lets a small number of experts govern a large dataset, and it depends on the routing being explicit rather than ad hoc. Review workflows that track agreement and route by confidence are what make the triage repeatable.

The related decision is which annotator profile fits which layer, which we looked at in more detail when discussing how to match a labeling workflow to a team.

Deciding, in practice

Before a project starts, run the non-expert instruction test on a sample. Build gold items with expert-verified answers whatever you conclude. Then measure both agreement and gold accuracy on a pilot batch.

High agreement with high gold accuracy means the crowd model fits. High agreement with low gold accuracy is the dangerous quadrant, and it means the task needed expertise that instructions did not supply. Low agreement means the rubric is underspecified, which is worth fixing before deciding anything about the pool.

That pilot costs a fraction of a full campaign and routinely changes how the campaign is staffed.

Recruit for the judgment, not the headcount

When the label is a judgment, the constraint is finding people who can make it and designing a workflow that spends their time well. HumanSignal Services recruits for niche expertise, designs the rubric and the review workflow, and delivers a verified dataset built to your spec. Book a scoping call to talk through which parts of your task genuinely need an expert.