Rubrics Design: The Missing Layer Between Guidelines and Good Annotations

When annotators disagree, the problem may be the reasoning process we designed for them, not the people doing the annotation.

We deliver evaluation datasets and human feedback at scale. The hardest part of that work is rarely the labeling itself. It’s calibrating what quality means. Clients arrive with dimension-based guidelines such as “evaluate accuracy, tone, and overall quality” that read clearly, but can produce low agreement when annotators apply them to real examples.

We’ve organically developed a rubric-building methodology to speed up that calibration and improve the consistency of its outcomes. The method translates broad evaluation requirements into an explicit sequence of observations and judgments, represented as a graph that makes the reasoning inspectable, and stress tested by synthetic annotators before the human pilot begins.

The missing layer between requirements and labels

Annotation projects usually begin with requirements written for project stakeholders. Those requirements describe what the final data should capture, such as quality, alignment, correctness, relevance, safety, or preference.

These concepts are useful for defining an objective, but they are rarely operational enough for an annotator. Two people can understand the phrase "high quality" and still apply different thresholds, attend to different evidence, or combine several defects into an overall score in different ways.

A common tactic is to expand the guidelines or add questions, but that does not necessarily reduce ambiguity, and can unintentionally add cognitive load and fatigue to annotators. You can collect more answers without making the relationship among those answers clear, and past a certain point, asking for more degrades the quality of everything you collect.

The ideal annotation task asks the minimum number of questions needed to extract the maximum relevant information. Rubric design is the process of translating initial requirements into an explicit model of how an annotator should reason from evidence to judgment, and is often the missing layer in the pipeline:

Requirements → reasoning model → annotation interface → labels

Each stage is only as good as the one before it. You can't make up for an under-reasoned rubric with a polished interface; you just collect unreliable labels more efficiently. This is also why the reasoning model should be testable on its own, before the interface is built and before human annotators are involved. The rest of this post describes how we structure that model as a graph and test it with synthetic annotators.

Representing the reasoning process as a graph

We have seen researchers manage this translation in spreadsheets, with questions arranged across tabs and formulas encoding the relationships among them. That approach can work, but it becomes complex and error-prone as the rubric grows.

To make that logic explicit and testable, we have built internal tooling that models the rubric as a directed graph. Each node contains a focused question, its possible answers, and the conditions that determine the next node. A path through the graph represents one evaluation trajectory.

Character continuity example

Consider an AI-generated video clip evaluated for character continuity: does a character’s appearance stay consistent across shots? The two example frames below illustrate a problem with asking that question on its own.

“Does the character’s appearance stay consistent across shots?” combines two decisions: whether the character is visible enough to compare, and whether their appearance matches. An annotator might answer “no” because the character visibly changed, or because the second shot is too dark to tell. The same label then describes two different failures: a continuity defect and a visibility problem.

That ambiguity can cause disagreement even when annotators see the same evidence. One may treat an obscured character as a failed continuity check; another may see no visible change and mark it as a pass. The question leaves them to decide what missing evidence means.

Split it into two questions, asked in order:

1. Is the character visible in at least two shots, with enough detail to compare?
2. If yes, is the character’s appearance consistent across those shots?

If the first answer is “no,” stop the continuity check and record insufficient visibility. In this example, the clip has a lighting problem; continuity cannot be assessed. Annotators no longer have to guess at a comparison the footage does not support.

The same structure applies to evaluations with more criteria. The separate example below shows how focused questions and conditional branches form a graph.

This structure provides several useful constraints:

• Each node should ask one question that an annotator can answer from the available evidence.
• A branch should appear only when it is relevant to a previous answer.
• A subjective judgment should occur after the workflow has collected the observations needed to support it.
• Every path should terminate in a meaningful outcome.

The graph is also a pre-interface artifact. Once its logic is stable, it maps naturally to a step-by-step annotation experience. The annotator does not need to see the full graph. They see the next relevant question, while the system maintains the state and path.

This separation matters. It lets us evaluate the reasoning independently from the presentation. We can ask whether a branch is missing before debating the placement of a control in the UI. The graph is also versionable: changes to questions, branches, and thresholds can be diffed, reviewed, and audited over time.

Collect evidence before asking for preference

One repeated source of disagreement is asking for an overall preference too early.

“Which image do you prefer?” may be the final question we care about, but it compresses many latent decisions into a single response. One annotator may prioritize instruction adherence. Another may prioritize visual coherence. A third may notice a factual defect that the others missed. Their answers differ, but the label does not explain why.

Product imagery example

Consider a request to generate a product image of a 12 fl oz sparkling-water can in a refrigerator, using supplied brand assets. The two outputs below fail in different ways.

Output A has an image defect: the can is too large relative to the surrounding food and containers. Output B fails to follow the branding instruction: it replaces the supplied mountain logo and wordmark with different lettering and artwork. An annotator who prioritizes brand fidelity may prefer A. Another who prioritizes realistic proportions may prefer B. A preference label alone does not tell us which failure drove the choice.

The graph lets us collect that evidence first:

• Does the can use the supplied logo and artwork?
• Is its size plausible relative to nearby objects?
• Are there other visible defects, such as unreadable text or distorted geometry?

These checks should be independent: an output can fail either one or both. Each observed issue can then activate a relevant branch for assessing severity. Only after recording those observations do we ask about acceptance or preference. The final judgment is still human, but we can see the evidence and tradeoffs behind it.

This distinction depends on the research objective. If we are measuring taste, disagreement is valid data. A preference study should preserve individual variation. If we are trying to determine whether an output satisfies a quality standard, unconstrained preference is usually a noisy proxy. Fact identification and preference ranking are different tasks and should be modeled as such.

Binary questions can create false confidence

Breaking a task into smaller questions helps only when those questions carry information. Replacing one broad judgment with a long sequence of A/B or true/false questions can create a different failure mode.

With two possible answers, some agreement occurs by chance. Raw agreement can also become inflated when a question strongly favors one response. If almost every item should be answered “yes,” multiple annotators may agree without demonstrating a shared understanding of the underlying criterion. Poor wording can produce a similar default-to-yes or default-to-no pattern.

The result can look consistent while adding little signal. At the level of an individual decision, repeated binary prompts can contribute something close to coin-flip noise. Across a workflow, that noise accumulates.

For that reason, we do not treat raw agreement as a sufficient measure of rubric quality. We also want to inspect:

• chance-corrected agreement;
• the answer distribution for each node;
• how often each branch activates;
• disagreement concentrated at particular nodes;
• correlations that suggest two questions may be redundant; and
• whether a node changes the final decision in a useful way.

Agreement is a diagnostic signal. Its interpretation depends on the structure that produced it.

Synthetic annotators provide a preflight test

Human pilots are necessary, but they are expensive and slow to repeat. They are not the ideal place to discover an obvious missing branch or a question that every annotator interprets differently.

The insight that changed how we approached iteration was that the graph could be executed by synthetic annotators. We can give several language models the same sample items, assign them slightly different annotator profiles, and run each one through every applicable node. The resulting traces can be compared in the same way that we compare human annotations.

The process looks like this:

1. Bootstrap a graph from the initial requirements and guidelines.
2. Select a realistic sample of evaluation items.
3. Run multiple synthetic annotators through the graph.
4. Measure agreement, answer distributions, branch frequency, and correlations by node.
5. Inspect disagreement hotspots and degenerate answer patterns.
6. Revise the question, answer space, or graph structure.
7. Repeat, then move the strongest version into a human pilot.

This is more useful than asking a model to critique the guidelines in the abstract. The model has to traverse the rubric against concrete items. A vague boundary becomes visible as disagreement at a node. An unreachable branch appears in the path frequencies. A binary question that always receives the same answer appears in its distribution.

The surprising part was not simply that synthetic annotators could complete the workflow. In our early experiments, changes that improved the synthetic results showed some correlation with better agreement when the rubric reached human annotators. That makes synthetic evaluation useful as a proxy for finding structural problems before paying the full cost in human time.

We would not use this correlation to claim that synthetic annotators predict human behavior in every domain. Models may share biases, collapse legitimate variation, or agree for reasons that people would not. The synthetic run is a preflight test, not a substitute for a human pilot. Its value is in rejecting weak rubric designs earlier and reserving human experiments for stronger candidates.

Rubric optimization is a human-AI workflow

The first graph can be bootstrapped from existing guidelines, but graph construction is not a fully automatic conversion problem. The difficult work is deciding what should be observed, which distinctions matter, where a concept should split into branches, and when the evidence supports a judgment.

AI assistance can make that work faster. A model can propose nodes, audit unclear scope, identify overlapping answer choices, or suggest a new branch based on disagreement traces. A researcher still decides whether the proposed structure represents the evaluation objective.

That division of labor is important. The model searches the design space and executes cheap tests. The researcher defines the semantics, evaluates the failure modes, and decides what evidence is sufficient. Human annotators then validate whether the resulting workflow functions under real conditions.

From a bespoke task to a repeatable method

Annotation workflows will remain domain-specific. Evaluating a generated image requires different evidence from checking a factual answer or reviewing a safety-sensitive interaction. Standardization should not mean forcing each task into the same questionnaire.

The repeatable part is the method:

• make the reasoning explicit;
• separate observation from preference;
• encode conditional paths;
• test individual nodes rather than only final labels;
• correct for chance and response bias;
• use synthetic runs to iterate cheaply; and
• validate the resulting workflow with people.

This gives us a more rigorous way to develop a custom annotation process. It also produces an artifact that can connect requirements, interface generation, evaluation traces, and later revisions. When production data reveals a weak node, we can locate it in the same graph and test a replacement.

Open questions

Several questions remain open. How reliably does synthetic disagreement predict human disagreement across domains? Which annotator profiles expose useful edge cases instead of adding artificial variation? Can we estimate the information contributed by each node? When should a graph preserve disagreement because the variation is the intended signal?

We are also interested in closing the loop between the graph and the interface. A validated reasoning model should be able to generate much of the annotation workflow directly. Production results should, in turn, point back to the nodes that need revision.

Better annotation depends on designing the path from evidence to judgment. A graph makes that path explicit. Synthetic testing lets us improve it earlier. Human annotators tell us whether it works.

If you are working on similar problems, we would like to compare notes.