Our Vision for the Future of Reliable Labeling Agents

Like many parts of the ML/AI pipeline, the use cases and approaches to Data Labeling have been rapidly evolving with the introduction of LLMs. In the last few months, a significant development in the dynamic world of AI caught our attention — the rise of agents (such as AutoGPT has shown us). We've delved deep into this topic on our blog and, simultaneously, embarked on an exploration to grasp the mechanics and potential of these agents applied to various data processing tasks, starting with the problem of data labeling.

Agents combine knowledge outputs from LLMs and action on them in production systems, thus their reliability to correctly and consistently perform operations is mission critical. As with many in industry attempting to use agents in real-world applications, we had the need to ensure reliability—beyond monitoring and observability—with the ability to guide agents with human feedback. We saw an opportunity to create a new agent framework that could dramatically increase the efficiency of data labeling (and broader application across data processing tasks), with the unique ability to be guided by human signal.

Meet Adala: Reliable Agent Framework for Data Processing

After significant effort to arrive at a successful prototype, we are excited to introduce Adala, the Autonomous DAta Labeling Agent. Upholding our open source principles, and encouraged by the large community that’s grown around Label Studio, we were committed to open sourcing the technology from the start.

Adala is a robust framework for implementing agents that specialize in advanced data processing tasks in data labeling and generation. Think of these agents as smart assistants with the novel ability to teach themselves and develop specific Skills, such as data classification, summarization, or data generation.

The agent’s learning is shaped by the context you set for it and the data they encounter. For Adala, you set this context by giving them a ground truth dataset—Label Studio is of course one option to build that ground truth dataset. As the environment evolves—perhaps with additional data or insights—agents further refine their skills. Skills are fundamentally versatile, adaptable, and composable, enabling agents to handle intricate tasks with ease.

The basis for this functionality derives from the agent Runtime – an LLM where the agent's code executes. We will support OpenAI to start, but we’ll be actively working to support more runtimes (ideally through contributions and collaborations).

To ensure our agents remember and build upon their experiences, Adala provides a Memory component—a dynamic storage space for the agent's acquired knowledge. For instance, retrieving the previous experiences of an agent’s errors (and subsequent human feedback) allows them a starting point from which to branch off into learning or improving skills.

Reliability in Agents Through Human Signal

To allow Adala to produce reliable agents, we devised two main strategies:

• Supervision Integration: Provide agents with 'ground truth data'—well-defined examples that serve as a learning foundation. This foundational data not only sets the learning parameters for the agent but also defines its operational environment.
• Constrained Generation: Ensuring that an agent's predictions are within a defined and bounded range of outputs.

Progress Through Open Source

Alongside our peers in the AI space, we believe that true progress in AI comes when knowledge is accessible and collaborative, and there’s a strong feedback loop. By releasing Adala with an OSI-approved open source license, we hope to inspire creativity and new applications we couldn’t have imagined, as well as drive important standards and best practices in a rapidly changing market.

We've designed Adala with modularity at its core, emphasizing our belief in strong contributions from the community. We eagerly invite the AI community to contribute by:

• Developing various agent skills and scaling up their reasoning abilities.
• Adding support for more runtimes and dataset formats.
• Creating new environments to capture ground truth feedback.
• Testing and improving the core software, examples, and docs.

What’s Next?

We believe Adala has the potential to reshape the landscape of data processing, training, fine-tuning AI models, and building AI applications. While the technology is early, we’re committed to helping our customers navigate the rapid pace of AI innovation and drive more efficiency into their data labeling processes.

Want to learn more?

We're hosting a livestream to show you how Adala applies Generative AI Agents to “act, observe, and adapt,” learning from human feedback while independently completing labeling tasks. Join us to learn more:

Tuesday, November 7, 2023
11AM Pacific / 2PM Eastern

Register now

This post was originally published on the HumanSignal blog, but we're republishing here to share with the greater Label Studio community.