How to collect preference data for creative and design models

You didn't spend six months building a generative design model only to watch it fail. A reward model that learned longer outputs are better doesn't help you. But that's what happens when you reach for the standard "A is better than B" interface on day one of preference collection. The annotation runs smoothly. The labels pile up. Then the model trains, and the outputs look exactly the same as before. The data reflected individual annotator taste, not a shared definition of aesthetic quality.

TL;DR

Binary "A vs. B" labels create low-signal pairs for aesthetic outputs without a rubric.

Define evaluation dimensions before opening the annotation interface.

Multi-attribute ratings outperform binary pairwise labels for reward model performance.

High annotator disagreement is diagnostic information, not noise to discard.

On-policy collection keeps labels useful as the model's output distribution shifts.

Why chatbot-style preference labels don't transfer to creative tasks

Pairwise preference collection was built for outputs with a discoverable correct answer. Does this response comply with the instruction? Is the factual claim accurate? Is the output safe? Creative outputs don't have correct answers. Two designers can look at the same two compositions and disagree, both reasonably and from genuine aesthetic judgment. When annotators apply a binary preference to a design output without shared criteria, the resulting labels map individual taste. The reward model can't generalize from that.

Low information content in each label causes more damage than disagreement alone. Preference datasets can be measured on three metrics: effective sample size, noise invariance, and information content (Apple ML Research). Pairwise comparisons score low on information content because a single binary choice doesn't tell the model which property of output A made it preferable. For a chat model evaluating helpfulness, that ambiguity is manageable. A design output can be style-adherent on one dimension and compositionally unbalanced on another. A single binary label can't capture that. For design outputs, it's nearly useless.

Define what 'better' means before you collect a single label

The rubric is the foundation of the dataset. Without it, you're collecting opinions. With it, you're collecting signal.

Break 'better' into ratable dimensions

For a design model, three dimensions cover most evaluation space:

Style adherence: does the output match the aesthetic direction in the brief? Color palette, visual language, typographic tone.

Compositional balance: is weight distributed in a way a trained designer would recognize as intentional?

Functional fit: would a designer ship this without structural revision?

Each dimension gets a numeric scale, a brief definition, and 2 anchor examples (one that scores low, one that scores high). The annotator rates each dimension independently before making an overall preference choice.

Why multi-attribute structure changes what the reward model learns

NVIDIA's HelpSteer3-Preference dataset (2025) used multi-attribute helpfulness ratings across 40,000-plus examples. The result was an 82.4 percent score on RM-Bench, well above what binary pairwise labels had produced. Multi-attribute ratings give the reward model a decomposed view of quality. It doesn't just learn that output A won. It learns that A scored higher on style adherence but lower on functional fit, and that the annotator weighted style more heavily for this brief.

Operationalizing the rubric in practice

Custom rubrics for post-training collection are the standard for specialized domains. HumanSignal's workflow for AI Research Labs runs pairwise ranking on domain-specific quality dimensions. Off-the-shelf rubrics produce off-policy signal that the model can't act on.

The rubric definition step takes a day or two. It also prevents weeks of re-annotation when you discover the labels don't correlate with the model behavior you wanted to change.

Structure the annotation interface for aesthetic judgment

A rubric only works if the interface makes it easy for annotators to apply it consistently. The wrong layout will undermine even a well-designed rubric.

Side-by-side with per-dimension ratings

Put both outputs on screen at the same time, with the rubric dimensions listed below each pair. The annotator scores each dimension for both outputs before selecting a winner. Forcing the attribute evaluation before the overall judgment stops the most common shortcut: picking the output that "looks better" and backfilling a reason.

For design outputs specifically, avoid a vertical stack layout. Designers read compositions horizontally. A side-by-side view mirrors how designers actually compare work.

Route disagreement to experts

When two annotators produce conflicting labels for the same pair, the natural instinct is to drop the pair or take a majority vote. Do neither.

High disagreement on a specific dimension tells you one of three things:

The dimension definition is ambiguous.

The annotator pool doesn't have the domain taste to evaluate that dimension consistently.

The model outputs genuinely occupy an aesthetic gray zone.

The first two require fixing the pipeline. The third is valuable training data. Route high-disagreement pairs to a domain expert for resolution. Preserve the disagreement metadata alongside the final label. That metadata gives the reward model information about where aesthetic judgment is genuinely contested. HumanSignal's consensus agreement framework treats disagreement as a signal that surfaces gaps in evaluation criteria rather than a failure to average away.

When the disagreement rate on a particular dimension stays consistently above 30 percent across a batch, that's your signal to sharpen the definition before continuing.

The throughput trade-off

Multi-attribute rubrics add 30 to 60 seconds per pair compared to binary comparison. For teams with annotation budgets under roughly $50,000 or models in early-stage fine-tuning, that overhead may not pay off immediately. A single focused dimension ("which output better matches the style brief") on a binary scale can produce adequate signal for a first training run. Add attribute dimensions once you have baseline model behavior to compare against. You'll know which properties to decompose because you'll see where the model diverges from what you wanted.

Evidence from structured annotation at scale

Sense Street builds generative language models for capital markets. They faced a similar challenge: highly specialized outputs, five languages, and domain-specific terminology that required consistent judgment across annotators. After implementing structured inter-annotator agreement and annotator-reviewer agreement workflows, they achieved a 150 percent increase in total labels and a 120 percent increase in annotations per labeler. Structured review raised throughput alongside quality, not instead of it.

Treat annotator disagreement as a quality signal, not noise

What high disagreement tells you

Three readings, three responses:

The rubric definition is ambiguous. When annotators disagree on a dimension but agree on the winner, the dimension is decoupled from actual judgment. Rewrite the definition and re-anchor examples.

The annotator pool lacks domain taste for that dimension. Disagreement clustering around specific annotators points to a calibration gap. Run a calibration session on reference pairs before the next batch.

The model outputs sit in genuine aesthetic gray zone. Disagreement spread across multiple annotators on specific pairs maps the edge of the aesthetic space. Label those pairs via expert resolution and keep the disagreement metadata.

Why low disagreement isn't automatically good

Artificially low disagreement is a warning sign. Noise invariance is one of three dataset health metrics. A dataset where annotators default to agreement scores lower on information content than one where genuine disagreement is recorded (Apple ML Research). A dataset where everyone agrees on every pair, for a subjective domain, likely reflects anchoring bias: annotators read each other's cues rather than forming independent evaluations. Blind annotation (annotators who can't see other labels before submitting their own) is the structural fix.

The disagreement data you preserve today becomes the calibration set for your next annotator cohort.

Feed preference data back into the training pipeline on-policy

Preference data has a shelf life. Labels collected from model outputs at version N lose value for training version N+2, because the model's output distribution shifts with each training run.

On-policy data, collected from the specific model being trained, matters more than pooled datasets (Nathan Lambert, RLHF Book). For a design model, that means pulling live model outputs regularly and routing them through the rubric-structured annotation workflow. Return labeled pairs to training on a cadence, not in a single initial batch.

The operational cost of that cadence is real, but it scales better than it looks. Geberit built an automated labeling pipeline with HumanSignal and saw 5x faster throughput, 95 percent annotation accuracy, and 4 to 5x cost savings compared to manual review (Geberit case study). Setting up the pipeline takes time upfront. Running it at cadence is cheaper than re-collecting a large batch after discovering the model has drifted past the labels you already have.

The rubric is the product, not the annotation run

Six weeks of failed reward modeling is rarely an annotator problem. The team that ran binary pairwise comparisons on aesthetic outputs didn't get low-quality data because the annotators were careless. They got low-signal data because no one defined what signal they were collecting.

Name your evaluation dimensions before you open the labeling interface. Every decision that follows (interface layout, disagreement routing, on-policy cadence) is an implementation of that naming. Teams that skip the rubric don't just collect noisier data. They collect data that can't be debugged, because there's no shared definition to compare labels against.

Define the dimensions. The rest of the pipeline follows from that single decision.