Can an LLM judge score creativity?

Human review of generated creative output is slow and expensive, so the obvious move is to put a model in the reviewer's seat and score everything automatically. On factual tasks that substitution has a clear validation path: run the judge against items with known answers and measure how often it gets them right. Creative work removes that path, because there is no known answer to measure against. What remains is agreement with human reviewers, and human reviewers on creative work disagree with each other by a wide and measurable margin. This piece works through what that leaves a judge able to settle.

Key takeaways

• A judge on a factual task is validated against a gold answer. Creative scoring has no gold answer, so the only available reference is human judgement, which itself disagrees substantially.
• In a HumanSignal benchmark across 7,817 segments, automated judges agreed with a human expert between 45.2% and 57.9% of the time, against roughly 72% for two experts with each other.
• On the DesignPref design comparisons the gap narrows without closing: a personalized judge reached 60.16% against a human pool whose own Krippendorff's alpha was 0.248.
• Judges already stand in for people more often than the literature admits. In a census of preference-data papers, 184 of 364 excluded records used only an automated scorer as their preference signal, and just 160 of 407 core papers re-validated a judge against fresh human judgement.
• Use a judge for triage, coverage, and regression detection against a fixed set. Do not use one to settle whether creative quality moved.

What validating a judge normally requires

The standard procedure has three parts: assemble items whose correct answers are established independently of the judge, run the judge, and measure agreement with those answers. If agreement is high enough for the decision at hand, the judge is fit for that purpose. The whole procedure rests on the first part, a reference standard that exists whether or not anyone runs a model against it.

This works for extraction, classification, transcription, and factual question answering. It also works for aspects of creative output that have objective components, such as whether a requested element is present or whether text is legible at a stated size. In those cases the judge is doing detection rather than evaluation.

Why creative scoring breaks that

There is no gold answer behind the judgement

When the question is which of two layouts reads better, there is no key. The human judgements are themselves the thing you are trying to capture, so grading a judge against them is not the same operation as grading it against the truth. Any accuracy figure here is really an agreement figure wearing different clothes, and reading it as accuracy overstates what has been established.

The human reference itself disagrees

The reference is not merely unavailable, it is internally divided. Research on designer preference data reports a Krippendorff's alpha around 0.248 across a pool of professional designers, with the disagreement treated as brand-specific judgement rather than annotation error. A judge cannot agree with a population that does not agree with itself. At best it can agree with a chosen aggregate, which is a narrower and more arbitrary target than "correct."

What the measurements show

Judges against a human expert

HumanSignal ran a quality estimation benchmark with Custom.MT across 7,817 translation segments, 17 language pairs, and 191 tasks, comparing six automated systems against professional linguists. The judges agreed with a human expert less often than the experts agreed with each other: between 45.2% and 57.9%, against roughly 72% linguist to linguist.

The per-system detail is more useful than the headline. Claude Sonnet 4 detected 88.3% of errors while capturing 27.8% of the available benefit, at 63.9% overall accuracy. Systems that captured more benefit let more errors through. That trade is the practical shape of judge deployment: you choose which error you would rather make, and no setting removes the choice. Translation quality is not design quality, and the specific figures will not transfer, but the ordering is the part that does.

Judges against a plural human population

Design comparisons show the same pattern with the numbers compressed. On the DesignPref data, the best personalized judge reached 60.16% while zero-shot GPT-5 reached 57.70%, against a designer pool at alpha 0.248. Personalization helped, which is itself informative: the judge improved by modeling a specific person's taste rather than by getting closer to a universal standard, because no universal standard was available to get closer to.

Read those together and the ceiling is visible. A judge scoring creative output performs in the high fifties to low sixties against a human reference, in a regime where two humans manage roughly seventy. That is useful, and it is not adjudication.

One caution about interpreting these figures. An agreement rate near sixty percent on a binary comparison is closer to chance than it first appears, since a coin reaches fifty. The margin a judge is adding over guessing is therefore small in absolute terms, and it shrinks further on the subset of comparisons where humans themselves split evenly. Those near-tie comparisons are also the ones teams most want resolved, which means judge performance is weakest precisely where the demand for it is strongest.

Where a judge is still worth running

Triage and coverage

A judge can score everything, which no human panel can. Use it to rank a large output pool so human attention lands on the items most likely to matter, and to maintain continuous coverage between human review cycles. Here a middling agreement rate is acceptable, because the cost of a mistake is a misordered queue rather than a wrong decision.

Regression detection against a fixed set

Run the judge repeatedly over an unchanged reference set and watch its scores over time. Because the set does not change, a shift in judge scores is evidence that something moved, even when the absolute scores are unreliable. This pairs directly with the work of deciding whether quality moved at all, where the judge supplies the alert and humans supply the verdict.

Criteria with objective components

Decompose your rubric and route the objective parts to the judge. Presence of a requested element, contrast ratios, text legibility, aspect conformance, and prompt adherence are all checkable. Reserve the criteria that carry taste for people. This is the most reliable value a judge delivers on creative work, and it comes from the decomposition rather than from the model.

Where it is not

Do not use a judge to settle whether creative quality regressed, to accept or reject work at a quality gate, or to generate preference data that trains a reward model without human re-validation.

The quality-gate case deserves its own warning. A gate applies a threshold to every item, which converts the judge's disagreement with humans into systematic rejection of a particular kind of work rather than into random noise. Whatever the judge undervalues gets filtered out consistently, and because the filtered items never reach a human, nobody observes the pattern. A ranking that is merely imprecise becomes a selection pressure once you put a cutoff on it.

That last one is where the field is drifting. In a HumanSignal census of preference-data literature, 184 of 364 excluded records carried only an off-the-shelf scorer or model judge as their "preference" signal, which is why they were excluded from a corpus about human preference. Within the core corpus, 76 of 407 papers mix model or synthetic labels into human preference data, and only 160 of 407 re-validate a judge against fresh human judgement. Training a model on judge output and then evaluating it with the same judge produces a number that rises while the thing it claims to measure does not.

How to deploy one without overtrusting it

Establish your human baseline first, because a judge cannot be interpreted without it. Have two qualified reviewers independently score the same batch, measure how far they agree, and treat that as the ceiling. Then measure the judge against the same batch. A judge below your human-to-human rate is an instrument for narrowing attention, not for deciding.

Control the known failure modes rather than assuming they are absent. Position, verbosity, and self-enhancement bias are documented behaviours of model judges, and the mitigations are procedural: randomize presentation order, blind the judge to which system produced an output, and avoid scoring a model with a judge from the same family. Recalibrate on a schedule, since provider-side model updates change judge behaviour without notice. Finally, keep the judge's scores separate from human scores in storage rather than merging them into one column, so you never lose track of which judgements a person made. The discipline of scoring outputs with no single right answer depends on that separation more than on any single model choice.

Run the judge and the human review in one loop

A judge earns its place when it widens coverage and a human settles the decisions that matter, which requires both to run against the same rubric and the same reference set. Label Studio Enterprise supports model-assisted scoring, per-criterion rubrics, multi-reviewer assignment, and agreement analytics in one workflow, so the judge's output and the human baseline stay comparable. Take a look at the platform if you are standing this up now.