How Yext Built an AI-in-the-Loop Benchmark with Label Studio Enterprise

Yext is a leading digital presence platform that helps brands manage and analyze how they appear across traditional and AI-based search, social media, maps, online directories, and other digital channels. The platform provides tools, insights, and recommendations powered by AI models and evaluation workflows that depend on high-quality labeled data.

Scaling that data pipeline is difficult. Manual annotation is expensive, time-consuming, and difficult to ramp during burst projects. LLMs offer a faster and lower-cost alternative, but introducing them into production workflows raises a different problem: how do you evaluate whether those labels are actually reliable?

To explore that question, Yext built an AI-in-the-loop annotation benchmark with Label Studio Enterprise. The project combined human annotation, adjudication, disagreement analysis, and LLM evaluation workflows inside a shared system to test how models could participate in large-scale marketing classification tasks alongside human annotators.

We needed another annotator, but hiring one for a 12,000-task burst project wasn’t realistic. So we asked: what if that annotator was an LLM?

The Challenge: Annotation at Scale Is a Bottleneck

The benchmark focused on 12,745 branded social media posts across six industry verticals and 1,414 businesses with 3,055 distinct locations. Each post was classified according to two taxonomies used for downstream marketing analysis: Funnel Stage (visibility, consideration, conversion, loyalty) and Intent (educational, promotional, engaging, inspirational).

Yext’s existing workflow used double annotation with adjudication:

1. Two human annotators independently labeled each task
2. Disagreements were routed to a third annotator, who consolidates both labels
3. Reviewed outputs became part of the gold dataset

The process produced reliable data, but it also created operational overhead during high-volume labeling projects. The team wanted to understand whether LLMs could reduce some of that workload while still producing trustworthy outputs.

The Plan: LLM as an annotator

The team set up a structured evaluation workflow to track and analyze disagreements across thousands of tasks. Using human-human (HH) disagreement rate as the baseline, they evaluated how well LLMs (GPT-5 and GPT-5 Mini using both zero-shot and few-shot prompting strategies) performed as “annotators.” They compared:

• Human-human disagreement: how often two trained annotators disagreed with each other on the same task
• Human-LLM disagreement: how often the model disagreed with an individual human annotator
• Gold-standard-versus-LLM disagreement: how often the model disagreed with the final adjudicated label after human review

They determined a baseline of 50.5% (disagreement rate between two trained human annotators), which served as a useful reminder that human agreement is itself an imperfect baseline, not a fixed standard of truth.

Label Studio Enterprise served as the shared environment for both the human annotation workflow and the LLM evaluation pipeline, allowing the team to create trusted gold datasets while comparing model outputs against the corresponding human labels within a single workflow.

Having the option to compare selected human annotations and gold labels with LLM predictions directly in the Label Studio platform was a true game-changer for the evaluation process.

That made it easier to review disagreements, trace annotation decisions over time, and analyze revision patterns across both humans and models. As it becomes more common to use AI to evaluate other AI systems, maintaining that level of measurement and traceability becomes increasingly important.

Key Findings: LLM Performance Approached Human Agreement Levels

The team’s findings show that across several classification tasks, disagreement rates between GPT-5 and human annotators were comparable to disagreement rates between two human annotators, particularly in the few-shot prompting setup.

Example disagreement scores for the GPT-5 few-shot model. (Few-shot prompting consistently improved alignment over zero-shot prompting across both taxonomies.)

Agreement rates only explained part of the story. The most useful insights came from analyzing where disagreements consistently appeared between humans and models.

For example, a few clear patterns emerged:

• Humans often tend to label Visibility and Conversion at the expense of under-labeling Consideration
• LLMs tended to over-assign Conversion instead of Consideration

These patterns helped the team refine annotation guidance, adjust prompts, and identify ambiguity within the taxonomy itself.

The Most Useful Signals Came From Disagreements

The team also conducted a qualitative review of disagreements on a sample of gold labels and LLM outputs to better understand why disagreements were happening.

For each disagreement, a senior annotation manager evaluated whether the mismatch:

• uncovered a potential omission or mistake in the human annotation
• represented a reasonable alternate interpretation of ambiguous content
• or reflected an unhelpful model misunderstanding

That review surfaced an important pattern: more than 50% of the disagreements either make sense or uncover a wrong label. In other words, disagreement was not always a signal that the model had failed. In many cases, it highlighted ambiguity in the taxonomy itself or exposed areas where humans didn’t follow the labeling guidance.

Rather than treating disagreement as a simple model failure, the team used it as a signal for improving prompts, labeling guidance, and benchmark quality overall.

Two AI-in-the-Loop Workflows Emerged

The findings ultimately pointed toward two practical ways to integrate LLMs into annotation workflows, depending on whether the priority was speed, cost reduction, or data quality.

1. LLM as Second Annotator

In this setup, the LLM replaces the second human annotator during primary labeling.

That reduces turnaround time and labeling costs while still maintaining structured review and adjudication workflows. In this project, that meant roughly $150 in LLM inference cost against an estimated $16,000 in human annotation cost saved, and ~15% reduction in overall project timeline*.

The tradeoff is that teams may lose some of the insights that come from investigating disagreements between two humans, which tend to be more informative and less schematic.

2. LLM as Quality Checker

In the second workflow, the human double-annotation process remains in place, but the LLM labels the adjudicated outputs afterward, acting de facto as a third annotator, assigned only to tasks where two annotators disagreed initially.

Seamless integration of AI labeling alongside human labeling allows the model to be employed as an additional reviewer that helps identify possible inconsistencies, omissions, or questionable labels in the ground truth dataset.

Unlike the “LLM as Second Annotator” workflow, this approach doesn't save money but rather adds costs from LLM inference and disagreement resolution time. However, the payoff here comes in increased trust and label quality.

*Note: Estimates assume a ~2-month project timeline for a 4-person part-time annotation team. Internal resource metrics are based on average US/EMEA mid-level data scientist hourly rates and order-of-magnitude effort estimates.

Both workflows point toward a broader shift in AI evaluation. Many teams are not removing humans entirely. They are restructuring how humans and models work together inside the evaluation pipeline.

The Workflow: Label Studio Enterprise for Evaluation

This work reflects a larger shift happening across AI evaluation workflows.

As AI evaluation expands with automation, new questions are raised about the trustworthiness of the evaluation process itself. How is quality measured? How are biases mitigated? How does the evaluation workflow adapt over time? The workflow surrounding the benchmark increasingly matters as much as the model itself. Its success is measured not just by the array of available functionalities, but also by the speed and ease with which they can be implemented.

Label Studio Enterprise is a structured environment for managing that workflow end-to-end, from human annotation and adjudication to LLM evaluation and disagreement analysis.

Structured evaluation is becoming increasingly important as enterprises move AI systems from experimentation into real production workflows. Benchmarking helps create that structure, but like any production system, it requires continuous review, calibration, and governance as models, prompts, and annotation guidance evolve over time.

From Static Benchmarks to Continuous Evaluation Workflows

Yext’s benchmark offers a more grounded view of how LLMs fit into annotation workflows.

The models helped reduce operational overhead and produced agreement rates close to human annotators in several areas. At the same time, the project showed that trustworthy evaluation depends heavily on the workflow surrounding the model, including adjudication, disagreement analysis, and prompt review.

One of the strongest findings was not that the models matched humans. It was those disagreements themselves that turned into useful signals for improving prompts, refining taxonomies, and strengthening the data quality workflow overall.

Interested in running a similar benchmark or building an AI-in-the-loop annotation workflow? Reach out to our team.

For detailed results and methodology from the Yext team, see the full research report here.