Why taste is a data problem: labeling for subjective quality

When annotators disagree on whether a response is "helpful" or "evasive," the default is to revise the guidelines. A longer rubric ships. Annotators are retrained. The model still doesn't converge. Teams treating subjective quality labeling as a description problem (one more word away from clarity) miss where the failure actually is. The problem is the absence of measurement infrastructure, and no rubric revision fixes that.

TL;DR

Annotation guidelines describe subjectivity. They don't measure it.

Annotator disagreement is signal; suppressing it before measuring it discards the information it carries.

Question-level agreement metrics reveal where judgment diverges, not just whether it does.

Pairwise ranking produces more consistent signal than absolute scoring.

LLM judges are stochastic; single-shot evaluations can't anchor reproducible quality measurement.

The guidelines trap

The industry reflex for a subjective labeling problem is documentation. A response marked "helpful" by one annotator and "evasive" by another triggers a rubric revision. The rubric gets longer. The failure repeats.

"Describe better" feels productive. If annotators disagree, the assumption is that they need a clearer model of what they're looking for. But two annotators can read the same definition of "helpful" and still apply different thresholds. One interprets it as technically complete. The other requires empathetic tone. No additional description closes that gap, because the gap isn't lexical. It's cognitive and contextual.

Description and measurement are different operations. Description tells an annotator what to look for. Measurement tells a team where judgment diverges and by how much. It also identifies whether that divergence is patterned or random. The field often conflates them. Poor data quality costs organizations an average of $12.9 million annually, according to Gartner. Subjective annotations drive much of that cost. Revising rubric prose produces better-documented disagreement. It does not reduce it.

Subjectivity is a signal, not a defect

Standard treatments for annotator disagreement on subjective tasks (adjudication, majority vote, reconciliation before labels enter training) discard information before it can be measured.

What disagreement actually tells you

iMerit defines Inter-Rater Consistency (IRC) as the degree of agreement among annotators labeling the same data. Its function: measuring how reliable the ground truth actually is, not whether individual labels are correct.

When two annotators diverge on "is this response culturally appropriate," that divergence carries information. Ground truth may not be achievable for this task from this annotator pool. Reconciling the labels before measuring the divergence erases that signal. The training data looks clean. The model trains on confident labels that were never agreed upon.

Why raw percentage agreement misleads

IRC measured with statistics like McDonald's omega accounts for agreement expected by chance, which percentage agreement ignores. If a task has three categories and annotators agree 60 percent of the time, chance alone might account for 33 percent. Omega-adjusted agreement can fall well below the surface number.

For subjective tasks, this gap is consequential. A 70 percent raw agreement rate on "helpfulness" can correspond to omega values that fall below any threshold justifying stable ground truth treatment. Teams shipping on raw percentage agreement are building on a foundation they haven't verified.

The scale problem makes this urgent

The labeling market is projected to reach $17.10 billion by 2030, growing at 28.4 percent annually. That growth stems from demand for training data where subjectivity dominates: preference labeling, RLHF, safety evaluation. The current volume of annotation in these categories is produced without the statistical infrastructure to verify whether any of it constitutes reliable ground truth.

Disagreement is not defect. It is the measurement instrument. The question is whether a team has built anything to read it.

Three tools that turn taste into measurable data

Question-level agreement, not task-level aggregates

Task-level agreement averages across every annotation decision in a task. If an annotator agrees with colleagues on tone, accuracy, and format but diverges on helpfulness, the aggregate score buries that divergence. Question-level agreement surfaces it.

For any task with multiple evaluation criteria, aggregate agreement offers limited diagnostic value. It tells you that annotators mostly agreed. It does not tell you where they diverged. The divergence itself reveals whether a quality dimension has stable ground truth. Measuring agreement at the question level isolates the specific criterion where human judgment splits.

Pairwise ranking over absolute scoring

AIxBlock's 2026 analysis frames the core issue clearly: RLHF doesn't assign objective labels. It encodes opinions about what is acceptable or useful in a specific context. Asking an annotator to rate a response 7 out of 10 on "helpfulness" requires anchoring to an absolute scale that exists only in their own head. Asking which of two responses is more helpful requires only a comparison they can reliably make.

Pairwise ranking reduces the cognitive load of subjective judgment. The resulting data is ordinal, reflecting the actual structure of human preference better than forced numerical scales. Pairwise comparisons produce more stable signal than the same annotators scoring on an absolute rubric.

Annotator calibration gates

Before production labeling begins, structured calibration measures individual annotator consistency on a set of known-disagreement tasks. Annotators who show high variance relative to the pool on specific quality dimensions get flagged, not penalized, but routed to calibration review. The goal is to separate annotators who diverge because their judgment differs from those who diverge because they misread the rubric.

Sense Street, a financial technology firm running trader chat analysis, applied structured inter-annotator agreement tracking. The goal: verify schema clarity and monitor individual annotator quality over time. The result: a 120 percent efficiency gain in annotations per labeler and a 150 percent increase in total labels generated. Annotators didn't work faster; measurement infrastructure eliminated the rework loop of catching disagreements after the fact. Calibration gates push that catch upstream, before disagreement compounds through review queues.

LLM-as-judge compounds the problem

Routing subjective evaluation to an LLM judge is appealing because human annotation at scale is slow and expensive. For low-stakes applications (iterating on chatbot tone, triaging early-stage prototypes) a single LLM pass with documented limitations may be a reasonable tradeoff. The cost of running multiple human evaluation samples can exceed the value of small reliability gains at that stage.

At higher stakes, the problem becomes structural. A 2025 reliability study from Northwestern University found that an LLM can be consistently wrong and still score as reliable. That's because reliability measures stability of judgment, not correctness. The two properties don't overlap. An LLM judge that consistently prefers verbose responses over concise ones scores high on internal consistency. It also mislabels quality in the direction of its own training biases. Single-shot evaluation cannot catch this, because there is nothing to compare the single shot against.

Because LLMs are stochastic, a single judgment is not a reproducible measurement. Multi-sample strategies are the minimum viable mitigation: run the same evaluation multiple times and measure consistency across runs using tools like McDonald's omega. Most teams don't apply them.

Mind Moves partnered with HumanSignal to evaluate a GenAI health assistant for a major health research institution. The team deployed 32 subject matter experts and 20 annotators. They worked across 20,000-plus tasks, assessing subjective criteria including interpretability and readability. The project produced a benchmark dataset for LLM-as-a-judge frameworks in clinical settings. As the Mind Moves team noted: "What we set out to do would have been impossible in Excel or JavaScript." The human pipeline served as the measurement instrument, not overhead.

LLMs offer speed. A measurement that cannot reproduce itself, though, is not a measurement.

When agreement is the answer, not the goal

Many labeling tasks have no single correct answer by definition. Whether a generated image is aesthetically appropriate, whether a support response sounds warm, whether a translated phrase carries the right cultural register: none have ground truth. They have distributions of human judgment, and those distributions are the data.

The LLM evaluation field is only now adopting these methods, as the statistical culture of objective labeling remains dominant. Subjective measurement is an established field. ITU-T Recommendation P.913 has described formalized methods for reaching consensus among multiple raters for video and audio quality assessment since 2021. The framework defines how to collect non-interactive subjective assessments. Applied consistently, it produces reproducible quality scores. The method characterizes where human perception converges and where it spreads. It does not produce a single right answer.

The ITU-T approach doesn't seek to eliminate rater variance. It models it. The output of a well-run subjective evaluation is a distribution: what percent of raters found this acceptable, where disagreement clusters, what conditions predict divergence.

Teams treating annotator disagreement as noise to be reconciled discard the most informative part of their data. Taste becomes data when the infrastructure is built to count disagreement, even if the rubric remains subjective.

The work is measuring disagreement, not explaining it away

Teams can describe what "helpful" means until every annotator has internalized the definition. The model's confusion persists anyway, because the rubric never measured where human judgment split or why. The missing ingredient was never a better word. It was an instrument pointed at the gap.

That is the harder claim this article makes: the information disagreement carries is more valuable than the consensus forced over it. Question-level agreement metrics, calibrated annotator pools, and consistency checks across multiple runs produce something no rubric revision can. They reveal where human judgment converges and where it legitimately doesn't. When annotators disagree on a subjective label, that disagreement is the measurement. Whether a pipeline is built to capture that variance or erase it determines whether the resulting training data reflects anything real.