How to build a labeling tool for rubric based candidate comparison

May 27, 2026

Evaluating foundation model outputs requires nuanced human judgment across multiple dimensions. When you benchmark different models or versions, you need a specialized interface that presents multiple responses side by side. This interface must capture specific dimensional scores and an overall preference without overwhelming the reviewer.

Configure a labeling interface that pairs multiple text objects with dedicated rating controls.

Generate the Extensible Markup Language (XML) layout automatically using a specialized coding agent.

Deploy the customized configuration and task data programmatically using the Label Studio Software Development Kit (SDK).

Measure inter-annotator agreement across both categorical rubric scores and ultimate winner selections.

Export the finalized comparison data as JavaScript Object Notation (JSON) payloads for model fine-tuning pipelines.

The problem

Building a custom application for rubric-based candidate comparison forces you to manage a highly specific and complex data shape. Your input data contains a single prompt and multiple long-form candidate responses that reviewers must read side by side while scrolling. Annotators struggle with context switching when they have to toggle between separate grading screens and the source text. Furthermore, routing internal model logs or third-party Application Programming Interface (API) outputs to a custom tool introduces severe compliance bottlenecks regarding data retention policies. Rebuilding this specialized side-by-side interface from scratch consumes months of engineering time and diverts resources from actual model evaluation.

The short answer

With Label Studio as the foundation, a coding agent can generate your exact labeling interface automatically. The agent uses two tools together: the XML labeling config builder skill produces optimized interface configurations from a plain-language spec, and the Label Studio SDK/CLI wires the config into a real project programmatically. Rather than building a new labeling application from scratch, agents generate the interface from your spec and deploy it into Label Studio in one pass.

Docs:

LLM-friendly docs (markdown) → https://labelstud.io/llms.txt

Pairwise comparison template → https://labelstud.io/templates/pairwise_comparison

Evaluate LLM responses tutorial → https://api.labelstud.io/tutorials/tutorials/evaluate-llm-responses

Task data format → https://labelstud.io/guide/tasks.html

What you're building

View the source prompt and multiple candidate responses in a side-by-side grid layout.

Score each individual candidate across predefined dimensions like helpfulness and safety using a star rating control.

Type detailed rationales for each dimension score into dedicated free-text input areas.

Compare the candidates directly to select an overall winner using a pairwise comparison picker.

Drag multiple candidates into ranked buckets for complex multi-way evaluation tasks.

Navigate rapidly through evaluation queues using built-in keyboard shortcuts.

How to build it in Label Studio

1. Set up the project

Start by installing Label Studio locally or deploying it on your own infrastructure if your rubric-based candidate comparison tasks involve strict compliance constraints regarding model outputs. One labeling unit for this task consists of a single text prompt and two or more generated candidate responses. You must format this input data as a JSON object where specific keys map directly to the corresponding interface controls. Include metadata fields like model versions or confidence scores in your data payload so reviewers can filter the task queue in the data manager. You can also preload external reference files like grading guidelines or ontology definitions to guide the evaluators.

2. Generate the labeling interface with the XML config skill

Hand the specification from the previous section to a coding agent running the XML labeling config builder skill. The agent processes your requirements and emits a validated Label Studio XML configuration that uses the precise tags required for rubric-based candidate comparison. This automated step ensures the interface layout properly binds every rating control to its corresponding candidate text.

<Text name="..." value="..."> – displays the source prompt and the individual candidate responses to the reviewer.

<Pairwise name="..." toName="..."> – collects the final side-by-side winner selection across two candidate objects.

<Rating name="..." toName="..."> – captures numerical dimension scores for a specific candidate text.

<TextArea name="..." toName="..."> – provides a space for annotators to type their rationale for a given score.

<Ranker name="..." toName="..."> – enables reviewers to drag and drop multiple candidates into a specific order.

3. Wire it into a project with the SDK

Direct your coding agent to use the Label Studio SDK/CLI to create a new project and inject the generated XML configuration. The agent can then upload your task data and import existing model predictions as pre-annotations to bootstrap the evaluation process. You can use this same agent loop to iterate rapidly on the configuration. Run a small batch of evaluations, watch the annotators struggle with the layout, ask the agent to regenerate the XML, and redeploy the updated interface.

4. Set up review and quality workflows

Establish a multi-annotator overlap strategy to ensure multiple human reviewers evaluate the exact same candidate pairs. You can configure a dedicated review stream queue for a senior evaluator to resolve disagreements when annotators pick different winners. For rubric-based candidate comparison, track inter-annotator agreement using exact match metrics for the categorical choices and numerical thresholds for the dimensional rating scores. These agreement statistics highlight ambiguous rubric definitions and help you identify reviewers who deviate from the baseline consensus.

5. Export and integrate

Extract the completed evaluations using the JSON export format as your default system of record. Downstream consumers of your rubric-based candidate comparison data will care most about the final winner selection fields and the individual dimension scores. Pass this structured payload directly to your machine learning training pipeline for reinforcement learning or load it into an analytics warehouse to benchmark different foundation models.

Why Label Studio for rubric-based candidate comparison

Render long-form text responses side by side using flexible grid layouts to eliminate annotator context switching.

Bind independent rating controls directly to specific candidate objects to manage complex nested data shapes.

Deploy the platform on local infrastructure to evaluate sensitive API outputs without violating external data retention policies.

Generate production interfaces instantly using the XML configuration standard to avoid months of custom engineering costs.

Sort the task queue by model confidence scores in the data manager to prioritize ambiguous evaluation pairs.

Common variations

Grade individual model responses against a rubric without directly comparing a second candidate.

Rate the semantic similarity of two generated text passages using numerical sliding scales.

Evaluate multi-turn chat threads to score the ongoing helpfulness of an artificial intelligence assistant.

Rank retrieved document contexts for relevance to support retrieval-augmented generation pipelines.

Next steps

XML labeling config builder skill → https://github.com/HumanSignal/create-xml-labeling-config-skill

Label Studio SDK/CLI → https://api.labelstud.io/api-reference/introduction/getting-started