logofirst
logofirst

Set up your labeling project

All labeling activities in Label Studio occur in the context of a project.

After you start Label Studio and create an account, create a project to start labeling your data.

  1. Create a project
  2. Import data.
  3. Select a template to configure the labeling interface for your dataset. Set up the labeling interface for your project.
  4. (Optional) Set up annotation settings for your project.
  5. (Optional, Label Studio Enterprise only) Set up review settings for your project.
  6. Publish your project. (Label Studio Enterprise only)

Create a project

When you’re creating a project, you can save your progress at any time. You don’t need to import your data and set up the labeling interface all at the same time, but you can.

  1. In the Label Studio UI, click Create Project.
  2. Type a project name and a description. If you want, choose a color for your project.
  3. If you’re ready to import your data, click Data Import and import data from the Label Studio UI. For details about import formats and data types, see Get data into Label Studio.
  4. If you’re ready to set up the labeling interface, click Labeling Setup and choose a template or create a custom configuration for labeling. See Set up the labeling interface for your project.
  5. When you’re done, click Save to save your project.

You can also create a project from a template by clicking Use Template. See more about project templates.

After you save a project, any other collaborator with access to the Label Studio instance can view your project, perform labeling, and make changes. To use role-based access control, you need to use Label Studio Enterprise Edition.

Delete tasks or annotations

If you have duplicate tasks, or want to remove annotations, you can delete tasks and annotations from Label Studio.

  1. In Label Studio UI, open the project you want to update.
  2. Filter the Data Manager page to show only the data you want to delete. For example, specific annotations, or tasks annotated by a specific annotator.
  3. Select the checkboxes for the tasks or annotations that you want to delete.
  4. Select the dropdown with the number of tasks, and choose Delete tasks or Delete annotations.
  5. Click Ok to confirm your action.

If you want to make changes to the labeling interface or perform a different type of data labeling, first select all the annotations for your dataset and delete the annotations.

Set up annotation settings for your project

Set up annotation settings to configure how you want annotators to perform labeling for your project.

Some annotation settings are only available in Label Studio Enterprise Edition. If you're using Label Studio Community Edition, see Label Studio Features to learn more.

Set up instructions for data labelers

In the project settings, you can add instructions and choose whether to show the instructions to annotators before they perform labeling.

  1. Within a project on the Label Studio UI, click Settings.
  2. Click Instructions, or in Label Studio Enterprise, click Annotation Settings.
  3. Type instructions and choose whether to show the instructions to annotators before labeling. If you want to provide additional details or links for reference, instructions support HTML markup.
  4. Click Save.
    Click the project name to return to the data manager view.

Annotators can view instructions at any time when labeling by clicking the (i) button from the labeling interface.

Set up task distribution for labeling

Select how you want to distribute tasks to annotators for labeling. Different from task sampling, use this setting to choose whether you need to assign annotators before they can start labeling.

  1. Within a project on the Label Studio UI, click Settings.
  2. Click Annotation Settings.
  3. Under Distribute Labeling Tasks, select one of the following:
    • Auto, the default option, to distribute tasks automatically to annotators.
    • Manual, to show tasks to assigned annotators first, then automatically distribute unassigned tasks.

Your changes save automatically.

You can’t assign annotators to tasks unless you select the Manual option.

Set minimum annotations per task

By default, each task only needs to be annotated by one annotator. If you want multiple annotators to be able to annotate tasks, set the Overlap of Annotations for a project in the project settings.

  1. Within a project on the Label Studio UI, click Settings.
  2. Click Annotation Settings.
  3. Under Overlap of Annotations, select the number of minimum annotations for a task.
  4. Choose whether to enforce the overlap for the default of 100% of tasks, or a smaller percentage.
  5. Choose whether to show tasks that require multiple annotations, tasks with overlap, before other tasks that need to be annotated.
  6. Your changes save automatically. Return to the Data Manager and assign annotators to the tasks so that they can annotate the tasks.

How task overlap works

For example, if you want all tasks to be annotated by at least 2 annotators:

If you want at least half of the tasks to be annotated by at least 3 people:

If you’re using manual distribution of tasks, annotators with tasks assigned to them label those tasks first, then Label Studio automatically distributes the remaining tasks to the project annotators so that the desired overlap and minimum number of annotations per task can be achieved.

Set annotating options

If you want, you can allow empty annotations.

  1. Within a project on the Label Studio UI, click Settings.
  2. Click Annotation Settings.
  3. Under Annotating Options, select Allow empty annotations. By default, empty annotations are allowed.

Set up task sampling

In Label Studio Community Edition, you can set up task sampling from the command line when you start Label Studio or from the Label Studio UI.

In Label Studio Enterprise, you can set up task sampling in the annotation settings for a project.

  1. Within a project on the Label Studio UI, click Settings.
  2. Click Annotation Settings.
  3. Select your preferred method of task sampling:
    • Uncertainty sampling, where tasks are shown to annotators according to the model uncertainty, or prediction scores.
    • Sequential sampling, the default, where tasks are shown to annotators in the same order that they appear on the Data Manager.
    • Uniform sampling, where tasks are shown to annotators in a random order.
  4. You can also choose whether to show tasks with ground truth labels first.
    Your changes save automatically.

Define the matching function for annotation statistics

Annotation statistics such as annotator consensus are calculated using a matching score. If you want the matching score to calculate matches by requiring exact matching choices, choose that option in the annotation settings. For more about matching scores and functions in Label Studio Enterprise, see Annotation statistics.

  1. Within a project on the Label Studio UI, click Settings.
  2. Click Annotation Settings.
  3. Under Matching Function, select Exact matching choices.
  4. For some types of labeling, you can also select a specific matching function.

Your changes save automatically.

Set up review settings for your project

Set up review settings to guide reviewers when they review annotated tasks. For more about reviewing annotations, see Review annotations in Label Studio

Review settings and the review stream are only available in Label Studio Enterprise Edition. If you're using Label Studio Community Edition, see Label Studio Features to learn more.

Set up instructions for task reviewers

In the project settings, you can add instructions and choose whether to show the instructions to reviewers before they start reviewing annotated tasks.

  1. Within a project on the Label Studio UI, click Settings.
  2. Click Review Settings.
  3. Type instructions and choose whether to show the instructions to reviewers before reviewing annotated tasks. If you want to provide additional details or links for reference, instructions support HTML markup.
  4. Click Save.
    Click Data Manager to return to the data manager view.

Set reviewing options

Configure the reviewing settings for your project.

  1. Within a project on the Label Studio UI, click Settings.
  2. Click Review Settings.
  3. Under Reviewing Options, choose whether to mark a task as reviewed if at least one annotation has been reviewed, or only after all annotations for a task have been processed.
  4. Under Reviewing Options, choose whether to anonymize annotators when reviewing tasks.
    Your changes save automatically.

Add members to a project

In Label Studio Enterprise, you can add members to a specific workspace or add members to a specific project within a workspace.

To add members to a specific project, do the following:

  1. Within a project, click Members and then click Manage Members.
  2. Locate the user that you want to add to the project.
  3. Select the checkbox next to the user’s name and click the > arrow so that they appear in the list of users that Belong to the Workspace.
  4. Click Save.

After adding a member to a project, you can assign them as a reviewer or assign them as an annotator to tasks in the project.

Publish project to annotators

In Label Studio Enterprise, you can hide projects from annotators so that you can fully configure the project before annotators can start labeling. When you’re ready for annotators to start labeling, publish the project to annotators.

Before you can open a project to annotators, make sure that you’ve done the following:

To publish a project, do the following:

  1. Open a project and navigate to the project Dashboard.
  2. Click Publish.
    After the project is published, annotators can view the project and start being assigned tasks according to the method that you use to distribute tasks for labeling.

Create a project template

If you want to easily create a project with the same labeling interface as an existing project in Label Studio Enterprise, create and use a project template.

  1. In the project that you want to use as a template, open the Settings.
  2. In the General tab for the project settings, click Save as Template.
  3. Add a title and description for the project.
  4. Click Save.

After you create a project template, you can use the template when you create a project.

  1. When viewing projects or workspaces, click Use Template.
  2. Select a template from the list.
  3. Type a project name and description for the new project.
  4. Click Create. The project is created and saved in your Sandbox.
    • Click Import to import data.
    • Click Data Manager to start annotating.

Where Label Studio stores your project data and configurations

All labeling activities in Label Studio occur in the context of a project.

Starting in version 1.0.0, Label Studio stores your project data and configurations in a SQLite database. You can choose to use PostgreSQL instead. See Set up database storage.

In versions of Label Studio earlier than 1.0.0, when you start Label Studio for the first time, it launches from a project directory that Label Studio creates, called ./my_project by default.

label-studio start ./my_project --init

Project directory structure

In versions of Label Studio earlier than 1.0.0, the project directory is structured as follows:

├── my_project
│   ├── config.json     // project settings
│   ├── tasks.json      // all imported tasks in a JSON dictionary: {task_id: task}
│   ├── config.xml      // labeling config for the current project
│   ├── completions     // directory with all completed annotations stored in one file for each task_id 
│   │   ├── <task_id>.json
│   ├── export          // stores archives with all results exported from Label Studio UI 
│   │   ├── 2020-03-06-15-23-47.zip

Warning: Modifying any of the internal project files is not recommended and can lead to unexpected behavior. Use the Label Studio UI or command line arguments (run label-studio start --help) to import tasks, export completed annotations, or to change label configurations.