What is aesthetic data, and how do you collect it?

Your model produces images that pass every technical check and a creative director rejects half of them. The information that would close that distance is a record of human judgement about quality, and most teams discover they do not have it in any form a model can consume. What they have instead is descriptive labels about content, a pile of thumbs-up votes with no structure, or a public preference dataset they cannot legally train on. This piece defines aesthetic data as a category, shows the evidence on why it stays thin where it matters most, and gives you a protocol to brief a collection against.

Key takeaways

• Aesthetic data records judgement about how good an output is, which makes it categorically different from labels describing what an output contains.
• The category is thinnest where it is most commercially useful. In a HumanSignal census of 407 core papers on preference data for creative generative models, design accounted for 20 and audio for 6, against 198 for image.
• Licensing is the binding constraint on reuse. Of 40 best-in-class datasets, 19 state no license at all and only 12 permit commercial use, so most public aesthetic data cannot lawfully train a shipped model.
• Instrument hygiene is mostly absent from published work: model blinding appeared in 16.7% of a random sample of 24 papers and attention checks in 8.3%. These are cheap to specify and rarely specified.
• Record who judged. Only 6 of 24 sampled papers linked judgements to a persistent annotator identity, and without it you cannot separate a house style from a population average.

What aesthetic data is, and what it is not

Aesthetic data is human judgement about the visual, stylistic, or craft quality of an output, recorded in a structure a model can learn from. The distinction that matters is between description and evaluation. "This image contains a chair" is a descriptive label, verifiable and reproducible. "The composition undercuts the focal point" is an evaluative judgement, and no amount of pixel inspection settles it.

That difference has practical consequences. Descriptive labeling has a right answer, which means it can be quality-controlled against gold items and scaled through consensus. Evaluative judgement has no external key, so subjective quality is a data problem before it is a modeling one, and the usual quality machinery does not transfer.

Aesthetic data is also broader than the preference pairs most teams start with. Preference judgements recorded by designers are one instrument within the category. Ratings against a rubric, ranked sets, marked regions, written critiques, and corrective edits all record aesthetic judgement, and they differ enormously in how much signal each judgement carries.

Why the category is thin where it matters

The published record shows a consistent shape. In a HumanSignal census of the literature on human preference data for creative generative models, covering 407 core papers, the modality counts were image 198, video 73, writing 33, speech 30, music 21, design 20, and audio 6. The corners where professional judgement is most expensive to source, design and audio, are the thinnest on the map. The field went where data was cheap to gather rather than where judgement was hard to replace.

That skew has a straightforward cause. Image preference data can be harvested from existing product telemetry and public voting interfaces, while design and audio judgement has to be commissioned from practitioners whose time is expensive. The literature accumulated where collection was cheap, which is also where the judgement is least differentiated. For a team trying to encode a specific standard, the published corpus is therefore least helpful in exactly the areas where a proprietary dataset would be most defensible.

Licensing compounds it. Reading the 40 best-in-class datasets as an upper bound on the field's behaviour, 19 state no license at all, 12 of those ship a live repository with no terms attached, and only 12 of the 40 permit commercial use. In the random sample of 24 that serves as the literature-wide estimate rather than the upper bound, a license was stated in 7 and commercial use allowed in 4. For a product team, a dataset you cannot lawfully train a shipped model on is not an asset, which means the practical supply is thinner still than the counts suggest.

The four decisions that determine whether it is usable

Volume is not what separates a usable aesthetic dataset from a decorative one. Four decisions made before collection do.

Who judges, and whether you record which one

Match the judging population to the judgement you need, since brand designers, illustrators, and motion designers do not share a standard. Then record annotator identity persistently. Only 6 of 24 randomly sampled papers linked judgements to a persistent identity, and 5 of 24 reported how many items each annotator judged. Without both, individual judgement is unrecoverable from the dataset, and you cannot tell whether a pattern reflects one strong opinion or a shared standard.

What the instrument asks for

A binary choice between two outputs is cheap, reliable, and carries almost no diagnostic information. Rubric scores across named criteria cost more per item and survive model changes, because they say which criterion moved. Richer instruments record reasons, marked regions, or corrective edits, and they are scarce: localized feedback appeared in 0 of the 24 sampled papers, and edits in 1. Building the scoring interface around the instrument you chose, rather than defaulting to a pairwise widget, is what makes the difference recoverable later.

Whether hygiene controls are in place

The controls that keep a judgement about the artifact from becoming a judgement about presentation order or model identity are cheap and usually skipped. Across the random sample of 24, model blinding appeared in 16.7%, order randomization in 29.2%, attention checks in 8.3%, permitted ties or neutral responses in 25%, any agreement statistic in 29.2%, and rater training in 45.8%. Read those as reporting rates rather than as proof of absence, since a control that went unreported may still have been applied. Either way, specifying them in your own protocol costs nothing.

How you aggregate, if at all

Collapsing several judgements into one number is the quietest decision in the pipeline and often the most damaging. Averaging across raters who genuinely disagree produces a score describing no actual person, and a model trained toward it optimizes for an output none of your reviewers would approve. Keeping per-reviewer labels preserves the option to model the distribution later, and you can always aggregate afterward, which is not true in reverse. This mechanism is what drives output convergence, as thin preference data narrows the range a model will produce.

A collection protocol you can brief against

Four stages, in order. Detailed guidance on running each lives in the collection protocol in detail; what follows is the shape to brief against.

Sampling

Choose comparisons that are informative rather than random. Oversample near-ties and the model's known failure modes, because comparisons with an obvious winner consume reviewer time and teach the model nothing it did not already encode.

Rubric

Decompose quality into named criteria before the interface exists. If you cannot name the criteria, reviewers will apply their own privately and you will be unable to diagnose disagreement afterward. Three to six criteria with worked examples at each score level is a workable starting point.

Calibration

Run small rounds before production volume, measure agreement per criterion, and tighten the criteria that produce the most scatter. Expect the rubric to change; that is the round doing its job. The practice of scoring outputs with no single right answer depends more on this stage than on reviewer seniority.

Audit

Gold answers do not exist here, so audit differently: use repeated items to measure within-reviewer consistency, attention checks to catch inattention, and agreement tracked over time to catch drift in the standard.

What to check before accepting a delivered dataset

Five things, all of which are expensive to add later. Per-reviewer labels rather than aggregates. Persistent reviewer identifiers with item counts. Criterion-level scores rather than a single verdict. A written record of the rubric version each judgement was made against. Explicit license terms permitting your intended use, including commercial training.

If a delivery is missing the first two, you have bought a population average you cannot decompose. If it is missing the last, you may have bought something you cannot lawfully ship.

Two checks are worth running on the content itself rather than the metadata. Re-score a sample of 30 to 50 items with your own reviewers and compare the distributions: a delivered set whose scores cluster far more tightly than yours has probably been aggregated or filtered toward consensus somewhere in the pipeline. Then look at how many items sit at the extremes of each criterion. A dataset with almost no low scores usually reflects sampling that avoided the model's failure modes, which is the sampling decision that most reduces the value of the result.

Brief a collection against your own criteria

Most aesthetic datasets fail on instrument design rather than on reviewer quality, and instrument design is decided before anyone starts labeling. HumanSignal Services runs collection across more than 5 modalities, 30 languages, and 50 knowledge domains, delivering per-reviewer, per-criterion judgement against a rubric your team defines. Book a scoping conversation to work through the criteria and coverage your case needs.