How to write an annotation spec for a domain you don't understand

You need labeled data for oncology triage notes, or derivatives contracts, or turbine vibration traces, and nobody on your team can read them. The obvious move is to hire someone who can, and it solves less than it appears to. The specialist you bring in still has to be told what to produce, and you still have to decide whether what comes back is usable. Both of those jobs stay with you, and neither requires domain knowledge to do well. What follows is how to write a specification precise enough to direct an expert you cannot second-guess, and how to design an audit that holds up when you cannot read the labels yourself.

Key takeaways

• In a study of human evaluation practice in natural language generation, under a third of papers released their guidelines at all, and 77.09% of the ones that did contained identifiable flaws. Assume your first specification is one of them.
• Two professional linguists independently scoring the same translation segments agreed on roughly 72% of them. Expert judgement has a noise floor, and a specification has to be measured against that floor rather than against perfect agreement.
• A specification that lists categories cannot be audited. One that attaches a disqualifying condition to every label can.
• You can validate work in a domain you do not understand by seeding items whose answers you already know and reading agreement patterns, rather than by reading the labels themselves.
• Calibration rounds on 50 to 100 items cost far less than relabeling a corpus.

Why hiring an expert does not remove the problem

Bringing in a specialist changes who applies judgement. It does not change who is accountable for defining the task, and the definition is the part that determines whether the output is consistent enough to train on. An expert given a category list will produce labels that reflect their own reading of the categories. A second expert will produce labels reflecting theirs. Both can be defensible and still disagree often enough to make the resulting dataset unusable as supervision.

That is not a hypothetical failure mode. A study of human evaluation practice in natural language generation found that only 29.84% of recent papers at top conferences release their evaluation guidelines at all, and that vulnerabilities appear in 77.09% of the guidelines that do get published, across a taxonomy of eight distinct flaw types. These are papers written by researchers who understood their own domains. The specification is a separate skill from the subject matter, and it fails often even in expert hands.

The staffing question is real, and there are cases where some data genuinely requires a specialist rather than a trained generalist. That decision is worth making deliberately, and it is a different decision from the one this piece covers. Here the assumption is that you have already concluded you need domain expertise you do not have in house.

Write the specification as a decision procedure

The useful shift is to stop writing definitions and start writing a procedure that resolves cases. A definition tells a reader what a label means. A procedure tells them what to do when two labels both look defensible, which is the situation where a specification earns its keep.

Start from the disagreements, not from the category list

Take 30 to 50 items that you already suspect are hard, hand them to two specialists independently, and collect the cases where they diverge. You do not need to understand the subject to see the divergence. Each disagreement is a place where your task definition is underspecified, and the specialists themselves can usually tell you what distinction they were weighing. Write that distinction down as a rule. Then repeat.

This inverts the usual order, where a specification is drafted first and disagreement is treated as an annotator problem. Drafting first means your document encodes the assumptions of whoever wrote it, and you have no way to find the assumptions that turned out to be wrong.

Give every label a disqualifying condition

For each label, write the condition under which it must not be applied even though it looks correct. A category with only inclusion criteria will absorb every ambiguous case, because applying it is always arguable. A category with an explicit exclusion forces a decision, and a forced decision is one you can audit later without knowing the subject. This is a structural property rather than a domain judgement, which is why it is available to you.

Record resolved edge cases as the specification grows

Keep a running list of decided cases with the reasoning attached, and treat it as part of the specification rather than as project notes. When a new annotator joins, the decided cases teach the task faster than the prose does. When a specialist proposes a change, the list shows what the change would break. Over a few rounds this becomes the most valuable artifact you own, and it is the one thing a vendor cannot hand you at the start.

Design an audit you can run without the domain knowledge

An audit that requires you to evaluate individual labels is closed to you. An audit built on structure is not, and structure is enough to catch the failures that matter.

Seed items whose answers you already know

Insert items into the queue where the correct label is established independently: cases adjudicated by two specialists who agreed, items with a documented outcome, or examples the specialists themselves supplied as unambiguous. You are not judging the label. You are checking whether the annotator reproduced a known answer. A seeded set of a few dozen items, refreshed periodically, gives you a per-annotator accuracy signal in a subject you cannot read. It also clarifies what counts as ground truth for the specific task, which is usually vaguer at the start than teams expect.

Read agreement patterns rather than individual labels

Route a fraction of items to more than one annotator and measure agreement across your annotators with a coefficient that accounts for chance, such as Krippendorff's alpha. The absolute value matters less than where the disagreement concentrates. Disagreement spread evenly across items suggests the task is genuinely hard. Disagreement clustered on one label, one annotator, or one document type points at a specification problem you can fix.

Calibrate your expectations against what expert agreement looks like when the work is being done well. In a benchmark HumanSignal ran with Custom.MT across 7,817 translation segments, two professional linguists scoring the same segments agreed on roughly 72% of them, and automated judges agreed with a linguist between 45.2% and 57.9% of the time. Translation quality is not your domain, and the specific figures will differ. The lesson transfers: skilled people applying real judgement to hard material do not agree at 95%, and a specification tuned to force that number is suppressing signal rather than producing quality.

Escalate on clusters, not on volume

Set the escalation rule before collection starts, and write it in terms of pattern rather than count. A single annotator diverging on one label across many documents is a training or specification issue. Many annotators diverging on the same small set of documents is a genuinely ambiguous subset that belongs in adjudication. Both are visible without domain knowledge, and both call for different responses.

Calibrate before you scale

Run the specification on a small batch, measure, revise, and run it again. Two or three rounds of 50 to 100 items will surface most of the definitional problems, and each round costs a fraction of what relabeling a full corpus costs after the fact. Practically, this means you run a calibration round before scaling rather than treating onboarding as a formality, and you accept that the first round will change the document.

The same structure tells you when your staffing model is wrong. If agreement stays low after the specification has stabilized, the task may need deeper expertise than the people applying it have, which is a signal about when a generalist reviewer stops being enough for the work in front of them.

What to hand a partner when you outsource the work

If the collection or annotation goes to an outside team, the specification and the audit design are what you are buying against. Hand over the decision procedure, the disqualifying conditions, the decided-cases list, the seeded items with their known answers, and the escalation rule. Withhold the answers to the seeded items. That package lets you hold a partner to a measurable standard in a subject you cannot evaluate directly, and it stays yours if you change partners later.

Bring in the expertise without giving up the specification

If the domain you need labeled is one you cannot staff, the specification and the audit are still the parts worth keeping in house. HumanSignal Services recruits from a network of more than 3 million experts across 50 or more knowledge domains, and works to a spec your team owns rather than one you inherit. Start a scoping conversation to walk through the task definition and the audit design for your case.