When generalist annotators aren't enough

Late 2025 brought a wave of AI lab announcements about pivoting to PhD-holding annotators. Much of the industry read this as confirmation that generalist annotators were becoming irrelevant. Most teams building production pipelines face a real cost floor, though. Domain experts don't scale to millions of labels. What those labs actually did was restructure the workflows around their annotators. The debate centers on how labels get validated, not simply on who submits them.

TL;DR

Generalists still outperform experts on value per label for a wide class of tasks.

Generalist annotation fails when teams skip evaluation gates, gold sets, and arbitration.

A tiered model (generalists for volume, experts for validation) outperforms either alone.

Sense Street's 60/40 annotator-to-linguist split produced 150% more total labels.

Fixing the workflow beats replacing the workforce.

Where generalists still outperform on value per label

The "death of the generalist" framing rests on the hardest cases: medical imaging, securities law, advanced mathematics. It ignores a large class of tasks where generalists remain the best option.

The 2026 AI Index records what researchers call the "jagged frontier" of model capability. Frontier models can solve PhD-level science problems and still fail at reading an analog clock. Generalist annotators are the right people to label these failures. They catch them not through specialized knowledge but through the unremarkable familiarity that expert annotators and model evaluators stop bringing to the work.

The economics reinforce this. Mid-level generalist annotators run $15–$35 per hour in 2026; domain experts in legal, medical, or technical fields run $40–$70. Any task requiring millions of labeled examples makes that gap decisive. General chat helpfulness, creative writing evaluation, UI and accessibility testing, and large-scale image classification all fall in that category. Annotating at that scale with experts isn't just slower; for most organizations, it's not financially possible.

Why generalist annotation breaks down

When generalist output falls short, teams often attribute it to annotator qualifications. Focusing on individual qualifications leaves the workflow design unexamined.

The International Telecommunication Union's annotation specifications (ITU-T DEL5.3) define a Standard Operating Procedure with four quality layers: labelers work independently, experts arbitrate conflicts, reviewers check output, and managers train and assess the team. Most commercial annotation programs implement one of those four. Some implement none.

When arbitration is absent, two annotators can disagree across a significant portion of a dataset and neither label gets flagged. When expert reviewing is skipped, systematic errors in the generalist pool propagate directly into training data. When annotator training and assessment don't exist, the program has no baseline to detect when a labeler's output has degraded.

The five pillars most teams skip

Industry benchmarks for annotation quality in 2026 identify five structural requirements for reliable labeled data: gold standards embedded in the annotation queue, expert-reviewed guidelines, consensus mechanisms for borderline cases, consistency tracking across sessions, and auditable output trails. None of these are about the annotator's background. All of them are infrastructure decisions made before a single label is submitted.

Without this infrastructure, it is difficult to know where quality issues originate. A pipeline with no quality signal produces unreliable output regardless of who fills it. The problem is measurement. Who fills the pipeline is secondary.

A scalable labeling program treats onboarding like quality control. Clear instructions ensure labelers interpret tasks the same way, guardrails catch low-effort behavior early, and evaluation gates ensure only those who score well can access production data. That sequence (instruction, guardrail, gate) is what most programs replace with a single PDF of annotation rules and a Slack channel for questions.

The tiered annotation model: how to combine generalists and experts

The binary framing of "cheap generalist or expensive expert" treats annotation as a commodity purchase rather than a system design problem. Teams that produce reliable labeled data at scale don't choose between generalists and experts. They assign them different jobs.

Defining the roles

The tiered model has two layers. Generalists handle the initial labeling pass. They work at volume, processing examples that are straightforward and don't require adjudication. Experts set the guidelines, review samples, adjudicate the borderline cases generalists flag, and validate output before it reaches training.

Tiered annotation is the structure that programs running at production scale have converged on. Expert time is a scarce resource. Deploying it on unambiguous examples (the 80 percent of a dataset where the correct label is obvious) wastes it. Expert time creates the most value when it's spent on the 20 percent of cases where judgment actually changes the label.

HumanSignal's annotator onboarding documentation makes the point directly: both doctorate-holding specialists and research-savvy generalists are needed to train models. The operational question is where each type of knowledge creates the most value in the pipeline.

What the structure produces

Sense Street, a company building generative language models for capital markets, used this structure in production. Their annotation team was 60 percent annotators and 40 percent linguists. The linguists didn't label alongside the annotators. They acted as gatekeepers, writing guidelines, mentoring annotators on edge cases, and reviewing output before it advanced. The results: 120 percent more annotations per labeler, a 400 percent increase in team size, and 150 percent more total labels.

The increased throughput results from linguists making the annotators' work more precise. When guidelines are specific and reviewers catch systematic errors early, labelers spend less time on re-work. More of their output passes quality checks on the first submission.

Where the tiered model has limits

The case for the tiered model doesn't extend to every validation layer in every domain.

A radiologist brings pattern recognition built across thousands of cases. A securities attorney applies statutory knowledge that can't be compressed into a guideline document. For final validation in genuinely high-stakes domains, no amount of workflow design compensates for absent domain knowledge.

The tiered model doesn't argue against specialists. It argues for deploying them where their time creates the most value: reviewing borderline cases, setting guidelines, and validating samples. They shouldn't be labeling the unambiguous examples that make up most of any dataset. The scope of their role narrows; the quality of their contribution increases.

Quality infrastructure that makes generalist pools reliable

Role definitions tell a team who does what. Quality infrastructure tells a team whether it's working.

Evaluation gates

Before any generalist annotator accesses production data, they should pass a quiz drawn from a gold set: a collection of examples with known correct labels. The gate has two functions. It screens out low-effort or low-comprehension workers before they touch training data. It also establishes a performance baseline for each annotator, allowing teams to detect degradation throughout the project lifecycle, not only at model evaluation.

Behavioral guardrails

Gold sets catch errors on the examples they contain. Behavioral guardrails catch the behavior patterns that produce errors at scale. Three patterns reliably indicate low signal: submissions completed too quickly to be read, duplicate responses across examples, and copy-paste behavior that bypasses the annotation interface. None of these always mean bad intent, but all of them produce unreliable labels. Flagging them early keeps low-signal submissions out of production data.

From infrastructure to outcome

Geberit scaled their labeling throughput 5x using HumanSignal's Prompts interface to integrate an automated labeling workflow. LLMs generated initial labels; human validators checked them against ground truth. The result: 95 percent annotation accuracy against ground truth and 4–5x cost savings compared to manual labeling. Quality infrastructure produced expert-level accuracy at generalist-level cost.

Foundation models are increasingly available to everyone. As HumanSignal has noted, the differentiator now is proprietary data and specialized knowledge from people. Infrastructure that extracts reliable signal from your annotator pool is what makes that data proprietary. Without it, you have labeled files. With it, you have ground truth.

The workflow is the answer

The shift toward PhD annotators was real. The assumption that generalists were finished did not follow from it. Labs that restructured didn't eliminate generalists; they built the quality infrastructure around them that most programs never had. The shift looked like a staffing change because the people in the seats were different. What changed was the system those people worked within: the tiers, gates, and guardrails that most annotation programs never build.

If your generalist labels are unreliable, the audit starts with the workflow. Where are the evaluation gates? Who reviews borderline cases? What does your gold set actually test? Replacing the workforce without answering those questions produces the same output with a higher payroll. Whether your labels are worth training on comes down to one structural choice: annotation as a commodity, or annotation as a designed system.