Blog

LLM-as-a-Judge promises speed, scale, and lower costs—but at what price? This post explores where automated evaluation works, where it fails, and how to make it trustworthy. From subtle biases to hybrid evaluation strategies, we break down how to evaluate the evaluator itself.

Krippendorff’s alpha is one of the most flexible ways to measure inter-annotator agreement. Unlike other metrics, it works with incomplete data and supports various label types. This guide walks through a real example—showing you how to calculate observed and expected agreement step by step, and what to do when scores are low.

The 1.18.0 release includes the much anticipated Dark Mode setting, the ability to export polygons in COCO format, and a bunch of usability improvements for annotators.

AI benchmarks are quietly failing us. Studies reveal that data leakage, leaderboard manipulation, and misaligned incentives are inflating model performance. This blog explores four pillars of reform, governance, transparency, broad-spectrum metrics, and oversight, and outlines how enterprises can build trust through a centralized benchmark management platform.

The latest In the Loop episode breaks down the Model Context Protocol (MCP)—a new standard for connecting LLMs to tools, data, and real-world actions. Learn how MCP enables practical, production-ready AI.

Not every ML team looks the same and your labeling workflow shouldn’t either. In this guide, we break down three common annotation team setups and how to tailor your tools and processes to boost quality, speed, and scale.

Learn how to generate synthetic Q&A pairs using Label Studio’s Prompts feature. This step-by-step guide walks through creating a project, writing prompts, and reviewing model-generated data—perfect for bootstrapping RAG systems or training LLMs when labeled data is limited.

Automated metrics can tell you what’s wrong with your RAG system—human review tells you why.
This post walks through a structured approach to evaluating RAG outputs using tools like Ragas and Label Studio. Learn how to prioritize weak responses, streamline human-in-the-loop review, and use feedback to iteratively improve your retrieval, prompts, and models.

Training a RAG system starts with clean, structured QA data—but messy source material makes that tough. In this post, we share how we used Label Studio and its Prompts feature to break down tasks, synthesize QA pairs, and build a reliable RAG assistant. If you're looking to scale RAG with better data and smarter workflows, this guide is for you.

This release includes a new home page, deep links, and personal access tokens for APIs. Learn about usability improvements and contributions from the community!

Using LLMs to label data is fast—but is it accurate? This post shows two ways to evaluate model performance in Label Studio, with or without ground truth. Learn how to compare outputs, track accuracy and cost, and choose the right model for your workflow.

In this post, we walk through a creative use case for Label Studio’s AI Assistant and Prompts: generating rich, image-based narratives. Follow along to see how you can build a project that combines structured labeling with a touch of storytelling.

When Stop the Traffik lost years of labeled data, they needed a faster, smarter way to rebuild. With Label Studio, they transformed their approach—bringing structure to messy reports, integrating AI for pre-labeling, and uncovering trafficking patterns hidden in plain sight.

RAG systems promise more accurate AI responses, but they often fall short due to retrieval errors, hallucinations, and incomplete answers. This post explores seven common RAG failures—from missing top-ranked documents to incorrect formatting—and provides practical solutions to improve retrieval accuracy, ranking, and response quality. Learn how to optimize your RAG system and ensure it delivers reliable, context-aware AI responses.

SmolDocling is designed for end-to-end document conversion, extracting text, tables, and layout with high efficiency. But how well does it perform on real documents? In this post, we walk through testing SmolDocling’s OCR capabilities using Label Studio and a step-by-step notebook to help you evaluate its accuracy.

Mistral OCR is setting a new standard for document understanding, but how well does it perform on your data? Using Label Studio, you can evaluate its accuracy, compare outputs, and fine-tune results. In this post, we walk through the process and share a hands-on handbook to get started.

RAG is transforming how businesses use AI, but without human oversight, its accuracy and reliability suffer. This blog explores the biggest challenges in RAG implementation and how human expertise improves data quality, retrieval relevance, and AI-driven decision-making.

Shamsuddeen Hassan Muhammad and his team are advancing African NLP by building language resources for low-resource languages like Hausa. With Label Studio, they scale their efforts to create high-quality datasets for sentiment analysis, hate speech detection, and emotion recognition, making AI more inclusive.

Explore the fundamentals of RAG, its advantages over fine-tuning, and the challenges of implementation.

We’re excited to announce the 1.16.0 release of Label Studio! In this release we have included a number of awesome new features, some critical security updates, and other improvements and bug fixes.

Learn about Reinforcement Learning with Verifiable Rewards, one of the leading training strategies for injecting learning signals into LLMs, successfully employed by models such as DeepSeek R1 and Tülu 3.

Learn the data curation and human supervision techniques that we believe are crucial to DeepSeek’s success by examining technical reports from DeepSeek-R1, DeepSeek-V3, and its predecessors.

Get started with content moderation labeling to classify potentially harmful data for training purposes, or evaluate the output of a model you already have in place.

Evaluating multi-turn conversations with Label Studio helps identify performance gaps, refine responses, and improve user experience for a more effective virtual assistant.

In this tutorial, we'll learn how to efficiently label medical images using Label Studio, from setting up your project to drawing bounding boxes and configuring classification tags.

Let’s dive into the highlights of 2024—from groundbreaking feature releases to engaging content that inspired and informed.

We’re excited to introduce Paginated Multi-Image Labeling! This new tag provides the fundamental tools for PDF labeling, which has been a popular request amongst our user community.

Understanding how your production model is performing is the most crucial yet most challenging step in the Machine Learning pipeline. This tutorial gives you an easy way to keep a pulse on what is really happening in production.

While data annotation for LLMs may look and feel somewhat different than the data annotation of the past, it’s still a crucial step of the machine learning process.

We’re excited to announce the 1.14.0 release of Label Studio! This release includes many improvements and a powerful new video annotation feature: Video Frame Classification.

Go through the entire fine-tuning process on OpenAI’s platform—from preparing recently-posted Wikipedia data to estimating costs and deploying your fine-tuned model.

Learn how Ameru is using Label Studio to power their smart garbage bins as they work to accelerate an economically sustainable zero-waste future.

Learn when you should use webhooks vs. the API in Label Studio, and see examples of what you can do with webhooks.

Learn how to use Ultralytics YOLOv8 object detection model with Label Studio.

OpenAI’s new Structured Outputs feature allows you to ensure outputs conform to a defined JSON structure. In this blog, we’ll explore how to leverage this feature for various labeling tasks.

We’ve released a new version of the Ultralytics YOLO ML backend connector designed for YOLOv8 and YOLO11, which now supports advanced object detection, segmentation, classification, and video object tracking with Label Studio.

Explore the topic of evaluation for LLMs, its importance, and how we should approach it. Learn how integrating systematic evaluations can help teams iteratively refine their models to meet real-world needs.

In this tutorial, we'll guide you through the process of setting up and using Label Studio in combination with Ragas (Retrieval-Augmented Generation Answer Scoring) and GPT-4 to build an optimized QA application.

Implementing RAG-based systems comes with challenges to be aware of, particularly in assessing the quality of generated responses. This article will walk you through some of those challenges.

Evaluate the output of LLMs and RAG pipelines with Label Studio using five new templates designed for human supervision of AI models.

Connect Segment Anything 2 (SAM2) with Label Studio to accelerate image and video data labeling.

We’re excited to announce the 1.13 release of Label Studio! This update includes a refreshed UI and some new Generative AI templates for you to use.

Delve into three effective methods to automate your labeling using Label Studio, including examples and resources.

In this article, we want to demonstrate a method of curating large datasets to reduce but not remove the cost for curating a high quality medical Q&A dataset in Label Studio and fine-tuning Llama 3 on this data.

This article is part of a longer series that will teach you how to develop and optimize a question answering (QA) system using Retrieval-Augmented Generation (RAG) architecture. In this tutorial, we are going to show you how to create a generator that builds responses based on those documents.

A new version the Label Studio SDK is now available with more functions, new docs, and a greatly improved developer experience.

An ongoing challenge for Large Language Models (LLMs) is their tendency to hallucinate. In this article, we explore four methods to automatically detect these errors.

Get the latest news on our upcoming event about AI Evaluation, new Discourse forums for community support, and tutorials including importing YOLO images and fine-tuning generalist models for NER.

In this introduction to our tutorial series on optimizing RAG pipelines, we'll introduce an example question answering (QA) system leveraging a Retrieval-Augmented Generation (RAG) architecture and outline three methods for optimizing your RAG pipeline utilizing Label Studio.

The short answer is: it depends. Read on as we explore this topic further, uncovering the advantages and drawbacks of each approach to help you make an informed decision.

Today, we're diving into Label Studio Format Converter, a handy tool that helps you encode labels into formats compatible with your favorite machine learning libraries, enhancing workflow efficiency.

Generalist models, like GLiNER, provide an excellent starting point for the tasks that they aim to solve. Fine-tuning these models offers us a way to improve their performance in the areas that we care about to solve business problems.

It's been another exciting month for the Label Studio Community. Read on to see what's been happening!

Different models are naturally going to excel at different tasks (just like humans). For users — especially those building products — having visibility into those tradeoffs is going to be a critical part of the decision-making process.

Sure, benchmarks are cool, but they don’t give you the feel or the intuition of how a model actually works. To get that, you’ve got to hack around with the model and throw real-world prompts at it — like you’d do in day-to-day tasks.

Harness GenAI and popular ML models for pre-labeling, interactive labeling, and model evaluation. Advancements to the ML backend and a new examples library makes it easy to create powerful automations for Label Studio workflows.

In the world of machine learning, data labeling plays a central role in training accurate and reliable models, and properly preparing your data for labeling sets the foundation for the success of your machine learning projects.

Learn how to build a prompt-centric workflow in Label Studio, which combines real-time prompt engineering with data labeling. The goal of this prompter is to make working with data and LLMs seamless and dynamic.

It's been another exciting month for the Label Studio Community. Read on to see what's been happening!

Announcing the latest Label Studio Release, 1.11.0!

With the latest Label Studio release, the Label Studio team is transitioning platform development from a multi-repository system to a monorepo structure.

Interactively Label Image-to-Text Datasets with Label Studio and Tesseract OCR.

These five tips for using Label Studio's API and SDK demonstrate these tools' powerful capabilities and flexibility for managing data labeling projects. From efficient project creation and task imports to advanced configurations and bulk data exports, Label Studio provides a comprehensive and streamlined approach suitable for beginners and advanced users.

Get a brief overview of RAG and how it relates to LLMs, learn when you might consider using RAG, and get a summary of some challenges based on current research you should be aware of should you choose to travel down this path.

Label Studio 1.10.1 is here, including several image labeling and security improvements.

Grounding Dino, Math Reasoning with Adala, and a recap of 2023!

Pair Grounding Dino or Grounding SAM models with Label Studio to automatically perform object identification and image segmentation tasks using text labels.

This example offers a glimpse into the practical utility and impact of Adala in data-guided automation and prompt tuning against a math-solving benchmark.

Messaging on the Label Studio Community Slack is currently unavailable. We are working to resolve the issue.

Label Studio 1.10.0 is here, featuring support for Large-Scale External Taxonomies (beta) and an update to the Ranker interface for fine-tuning LLMs!

Learn how to build an image annotation workflow using Label Studio and Segment Anything in Hugging Face Spaces!

The latest release of Adala, an agent framework to automate data processing tasks, including data labeling and data generation, features more ways to capture feedback and parallel skills execution contributed by a community member.

With the rise of deep learning and machine learning techniques, computer vision has become essential in various applications. But how do you get that data into your AI model effectively?

Introducing a new open source agent framework that can dramatically increase the efficiency of data labeling (and broader application across data processing tasks), with the unique ability to be guided by human signal: Adala.

Exploring how LLMs and agents can be leveraged to transform the annotation workflow, aiming for greater data labeling efficiency, accuracy, and scalability.

The Label Studio community is excited to announce the 1.9.1 release of Label Studio.

With their dynamic capabilities, agents can potentially adapt to the nuances of different datasets and environments, ensuring that the labeling process remains efficient regardless of the data's nature.

It's been another exciting month for the Label Studio Community. Read on to see what's been happening!

We’re excited to announce the latest version of Label Studio, which includes labeling configuration autocomplete.

Dive into the top 5 open dataset resources that could be the key to your next project's success

Our Data Scientist in Residence, Jimmy Whitaker, recently outlined how to fine-tune foundation models for AI applications. In case you missed it, here's a quick summary of the workshop.

We recently held a workshop demonstrating the powerful integration between the Segment Anything Model (SAM) and Label Studio. If you missed it, you can get the recap right here.

It's been another exciting month for the Label Studio Community. Read on to see what's been happening!

We're excited to announce the launch of our latest version of Label Studio, which features contextual scrolling!

Learn how to connect the object selection power of Segment Anything to the annotation power of Label Studio.

Hugging Face is a highly esteemed platform widely recognized as one of the most valuable resources for discovering and utilizing Language Model (LLM) technology.

We explore how Label Studio can be employed to build a Question-Answering bot trained on domain-specific knowledge.

Learn the lingo around fine-tuning LLMs. We've compiled a comprehensive glossary of terms that you should know when embarking on your LLM tuning journey.

With the latest 1.8 release — we launched the all-new Generative AI template library — a set of templates designed for you to work with foundation models and adapt them to your needs, right within Label Studio. Get started with fine-tuning models such as Llama 2, GPT-4, Whisper, Alpaca, and PaLM 2 among others.

Out of the box, LLMs can handle general tasks fairly easily. But what fine-tuning methodology should you use when you need to complete tasks that require more specialized knowledge than what LLMs typically have?

It's been another exciting month for the Label Studio Community. Read on to see what's been happening!

With just a few quick configuration changes, you can launch a compact Label Studio production environment for your annotation and labeling projects, all within Hugging Face.

Machine learning models are the race cars of artificial intelligence, designed to deliver fast and accurate results. But just like a race car, they need the right fuel - in this case, high-quality data that has been appropriately prepared.

Fine-tuning plays a pivotal role in harnessing LLMs for specific tasks. By exploring the process of fine-tuning and understanding its importance in LLMs, you will gain the knowledge and practical skills needed to optimize LLMs.

Learn how to automate labeling tasks using GPT-4 with the Label Studio Machine Learning backend.

Learn some advanced techniques to get the most out of Label Studio and drive more efficiencies into your labeling process.

Hundreds of thousands of open source LLMs are available, and choosing one is not easy. However, there are a small number of highly capable models that we think are excellent choices today.

It's been another exciting month for the Label Studio Community. Read on to see what's been happening!

Ever wonder how our JSON format is generated or how to structure your JSON files to import data, pre-annotate your datasets, or work with the ML backend? Wait no further! We've got you covered with this blog post.

HumanSignal is about the signal that humans provide to models, helping them to adapt, learn, and align with the needs of organizations and society at large.

We're excited to announce the latest version of Label Studio, specifically designed to help create datasets for fine-tuning Large Language Models (LLM) like ChatGPT or LLaMA.

Segment Anything is a popular model you can use to accelerate computer vision & image labeling. Learn how to use the Segment Anything integration with Label Studio.

It's been another exciting month for the Label Studio Community. Read on to see what's been happening!

Learn the background knowledge and steps required to integrate Machine Learning models into your Label Studio workflow.

How to use OpenMMLab with Label Studio's machine learning API to provide semi-automated bounded-box labeling using the mmdetection library.

Visit discuss.labelstud.io to see solutions to frequently asked questions, tips and tricks, and helpful discussions across the Label Studio Community. These have all been logged with the help of a new Archivist Bot, which will help log popular threads from within the existing community Slack.

RLHF has enabled language models trained on a general corpus of text data to be aligned with complex human values. This article details how you can train a reward model for RLHF on your own data.

In this article we explore the profound impact of human feedback on AI and examine how powerful tools like Label Studio are catalyzing a paradigm shift in AI training methodologies.

To help you get started with data labeling, we've created this guide that outlines the best practices to ensure you label your data accurately.

We are excited to announce the release of Label Studio 1.7.3, which brings a host of new features and improvements to our open-source data labeling tool.

It's been another exciting month for the Label Studio Community. Read on to see what's been happening!

How do you identify and maintain high-quality data when you’re knee-deep in a project? And what does “data quality” mean in the context of machine learning? Read more to find out!

Heartex co-founder and CEO Michael Malyuk recently did an AMA over at BentoML’s Slack community to talk through a variety of questions around data labeling, MLOps, and workflows that combine the two.

In this article, we’ll explore what object detection is, the challenges involved in implementing it, and share a step-by-step breakdown of getting started with this exciting technique.

Vikram Chatterji, co-founder of ML data intelligence company Galileo just published a framework for massively increasing the velocity of your ML workflow using a tight integration between Label Studio and Galileo.

It’s been a big month for the Label Studio community, full of releases and collaborations with the open source ML community. Here’s a quick recap of the flurry of announcements from this past month, including one that includes some magic that could have a big impact on your annotation workflow!

Join the Label Studio Community Team for our Community Town Hall Series on April 13th at 12PM ET. We’ll be sharing what’s happening in and around the community including future plans, our latest release, and more!

Evaluating GPT-4’s release through the lens of open source data science leads us to see the possibility and potential for long-term impact — but it also reiterates the underlying importance of dataset development and context provided by human feedback through processes such as reinforcement learning through human feedback (RLHF) or human-in-the-loop processes.

Label Studio 1.7.2 introduces a new Magic Wand tool for image annotation and significant updates to our audio labeling functionality.

Check out the new Label Studio Integrations Directory to explore a growing number of integrations and resources, including documentation and links.

This tutorial is designed to get you started with data annotation using Label Studio inside of Hugging Face Spaces.

A complete guide on how to build integrations for Label Studio, including database, storage, machine learning, web front end, and platform integrations.

The Label Studio docs have a new look; with cleaner information architecture, and more ways to contribute.

The Pachyderm integration with Label Studio makes incorporating data versioning into any data team’s labeling process easier.

Read step-by-step instructions to integrate Label Studio and Pachyderm, including code examples.

Audio labeling tutorial, new academic program, plans for PyCon & PyData Berlin in April, and more!

In this guide, you'll learn how to set up an audio labeling project in Label Studio. Then we'll walk you through importing and labeling your data. You'll also see the many different features in Label Studio's audio labeling interface.

Learn more about recent advances in RLHF, one of the essential training methods for building large generative models like ChatGPT.

Brand new to Label Studio? We’ve created a jam-packed new tutorial with the most important information to get you up and running.

Lots happening in the Label Studio community this month, check out these great resources and opportunities.

Do you want to share your knowledge with the community? We’d love for you to contribute content!

How TotalEnergies gave their oil rig engineers a cutting-edge tool for predicting and preventing critical equipment failures.

The latest version of Label Studio is here. Significant features include performance improvements to its audio labeling engine and new support for production-ready deployment methods using Terraform scripts or Helm charts.

The 2022 Label Studio Community Survey shows how data teams are increasing investment in data preparation and management to achieve better ML/AI outcomes.

A machine learning model can send unintended, dangerous conversational responses. Train your models to avoid these outcomes by learning what machine learning bias is and how to minimize it.

With this guide, learn how audio classification works and how to implement it when building audio ML projects, so you can optimize your ML models and build a better overall product.

This article takes a look at audio classification and discusses the various use cases that can benefit from this technique.

When choosing an annotation tool for your audio classification project, you need to carefully study its unique features and ensure that it works well with the rest of your stack. Finding the right tool will give you the best value in your audio classification projects.

In this guide, we dig into image classification—what it means, how it works, and the main steps to help you get started.

Instituting an efficient data labeling process is the key to eliminating inaccuracies in the data fed to machine learning models. Here are some generally applicable principles that can improve the efficiency and accuracy of your data labeling process.

Some developers and data scientists just want to grab code, download a repo and go. If that’s your style, choosing a fully-featured open source sentiment tool might be right choice for you.

The Label Studio user community survey is now live! Your feedback is critical to improving the product and helping bring new users into the community.

Label Studio 1.6 introduces a new video player that allows for frame-by-frame video object tracking, plus a major update to the annotation UI for improved ergonomics and efficiency.

Learn six of the most common data labeling mistakes we see in ML projects and the fixes that can help you maintain consistent, accurate training data.

From time series to active learning to the Python SDK, check out the top 10 features on the path to 10,000 GitHub stars and growing!

This blog covers the basics of sentiment analysis: key components, use cases, challenges, and solutions.

Sentiment analysis provides consistent, data-driven insights for businesses looking to better understand their users and products. Here’s how to get started.

Release notes and information about Label Studio version 1.5.0.

See what data labeling can learn from opossums and what Heidi means to Label Studio.

Release notes and information about Label Studio version 1.4.0, announcing the Label Studio SDK to work with the open source data labeling tool Label Studio, or the enterprise version Label Studio Enterprise.

Introducing the community reward program for the Label Studio open source data labeling tool!

Learn how various types of data drift and how data drift impacts model performance, along with several examples of how data labeling can tackle data drift.

Release notes and information about Label Studio version 1.3.0, featuring ML-assisted labeling.

Use webhooks from open source data labeling software Label Studio to seamlessly integrate your Amazon SageMaker model development workflow for your machine learning and data science projects using AWS Lambda, Amazon S3, and public domain bird images.

Use open source data labeling software Label Studio to quickly create YOLO v3 and v4 compatible datasets for training purposes and image labels for image object detection data science and machine learning projects.

Release notes and information about Label Studio version 1.2.0, featuring event webhook functionality for annotations and tasks.

Use open source data labeling software Label Studio to improve optical character recognition (OCR) results for receipts, invoices, menus, signs, and other important images processed with Tesseract and Python.

Release notes and information about Label Studio version 1.1.0, with improved data labeling functionality for image annotations and object character recognition (OCR) labeling for machine learning projects.

The top 10 important considerations for NLP labeling and functionality in labeling tools for natural language processing machine learning projects.

Use open source data labeling software Label Studio to improve audio transcriptions of customer support calls, video conference meetings, and other audio recordings.

Use Label Studio to evaluate named entity recognition parsers like spaCy and revise predictions by annotating a gold standard dataset for your data science and machine learning projects.

Label Studio Release 1.0.0 rewrites Label Studio to support multiple users, projects, and scalable data labeling and annotation for machine learning and data science projects.

Label Studio Release 0.9.0 includes improvements to data management to help you explore and improve your datasets for machine learning projects.

Label Studio Release 0.8.0 includes updates to support data labeling for time series data.

Label Studio Release 0.7.0 includes new connectors to integrate Label Studio with cloud storage, including Amazon AWS S3 and Google Cloud Storage.

Label Studio Release 0.6.0 includes nested data labeling, per-region labeling, updates to machine learning backend integration, filtering, and more.

Label Studio Release 0.5.0 includes image segmentation, relations labeling, named entity recognition performance, image ellipses labeling, and more.