logofirst
logofirst

Review annotations in Label Studio

Beta documentation: Label Studio Enterprise v2.0.0 is currently in Beta. As a result, this documentation might not reflect the current functionality of the product.

After multiple labelers have annotated tasks, review their output to validate the quality of the results. You can also perform this task after a model has predicted labels for tasks in your dataset. To configure the settings for reviewing annotations, see Set up review settings for your project.

The annotation review workflow is only available in Label Studio Enterprise Edition. If you're using Label Studio Community Edition, see Label Studio Features to learn more.

Why review annotations?

Data labeling is a crucial step for training many machine learning models, and it’s essential to review annotations to make sure that only the highest quality data is used to train your machine learning model. If you don’t review the quality of labeled data, weak annotations might be used when training your model and degrade overall model performance.

Review annotated tasks

After you assign reviewers to tasks, they can review annotated tasks. Administrators and project managers can review tasks at any time, without being added to a project.

  1. Reviewers can click Review Annotations for a specific project, then click Review All Tasks on the Data Manager to start reviewing tasks. Administrators and project managers can click Explore All Reviews from the Data Manager or Explore Review from the Dashboard to review tasks.
  2. Review the first task and annotation. By default, you view the tasks in numeric order. If you want to change the order that you review tasks, see Choose what to review. You can see the annotator and their annotation.
  1. Continue reviewing annotated tasks until you’ve reviewed all annotated tasks. Click Data Manager to return to the list of tasks for the project.

If there are multiple annotations, you can select the tab of each annotation by annotator and result ID to view them separately. The annotation result ID is different from the task ID visible in the left menu. To see annotations side-by-side, you can click the task in the Data Manager and view a grid of annotations in the task preview mode.

Choose what to review

You can review tasks in random order, or order tasks in the project data manager in different ways, depending on your use case:

Assign reviewers to tasks

As an administrator or project manager, you can assign reviewers to tasks, or people with access can review tasks on an ad hoc basis. Anyone who is assigned to a task or who completes a review of a task appears in the Reviewers column on the Data Manager. You must first add a reviewer to the project or add members to the project workspace before you can assign them as a reviewer.

  1. For a specific project, select tasks on the Data Manager.
  2. Select the tasks dropdown and select Assign Reviewers.
  3. Select names of reviewers and click the > arrow to assign them to the selected tasks.
  4. Click Assign.

You can assign reviewers to multiple tasks at once, but you cannot remove reviewers from multiple tasks at once.

Review annotator activity on the project dashboard

Use the project dashboard to review annotator activity. For a project, click Dashboard to view the dashboard.

If you don’t see an annotator’s activity reflected on the dashboard, make sure they have been added as a member to the project.

Review dataset progress

The dataset progress displays the number of tasks considered to be fully annotated for the project. If the project requires a minimum annotation per task of more than one, some tasks might not appear as “annotated” because they are not yet fully annotated by the project standards.

You can review how many tasks remain to be completed by annotators, how many tasks have been skipped, and how many tasks have been reviewed.

Review label distribution

For specific labels, you can see in a donut chart how many labels of each type were applied to the tasks. Use this chart to identify possible problems with your dataset distribution, if some labels are overrepresented in an annotated dataset compared with others.

For example, if you’re developing a dataset of OCR images, and 90% of your tasks have Text labels and 10% have Handwriting labels, you might want to increase the number of images of handwriting in your dataset, to improve the eventual accuracy of a machine learning model trained on this dataset.

Verify model and annotator performance

To verify the performance of specific annotators, review the Members section for a specific project. If you don’t see an annotator’s activity reflected, make sure they have been added as a member to the project.

Review annotator performance

For each project, you can review the project dashboard and review the Annotator Performance section to learn more about the annotators and their annotations, as well as overall annotator consensus.

Discover how many annotators have worked on the project, and how many hours they cumulatively spent labeling. You can also see the total number of annotations produced by the annotators, separate from the total number of tasks in the project.

Review a table to see the following for each annotator:

Review annotator agreement matrix

You can also review the overall annotator agreement on a more individual basis with the annotator agreement matrix.

Review the annotator agreement matrix to understand which annotator’s annotations consistently agree with or don’t agree with other annotator’s annotations. You can also filter the matrix to show specific agreement statistics for each label, or view the Overall agreement matrix. See more about how annotator agreement is calculated.

To see the specific annotations contributing to the agreement, do the following:

  1. Open the Data Manager for the project.
  2. Locate a task annotated by the different annotators that you want to compare.
  3. Click the task to open the task preview.
  4. Click each annotation tab to compare how the different annotations differ. The initials of each annotator appears in the tab header with the annotation ID.

Review agreement distribution across tasks

You can also review the distribution of agreement percentages across project tasks. A bar chart depicts the number of tasks with a specific agreement percentage. The more tasks with higher agreement, the higher quality your dataset is likely to be. Clusters of agreement percentages for specific tasks might mean that some tasks are easier to label than others, while other tasks are more confusing and difficult to label consistently.

Review annotations against ground truth annotations

Define ground truth annotations in a Label Studio project. Use ground truth annotations to assess the quality of your annotated dataset. Review ground truths to make sure that annotators are accurately labeling data at the start of the project, and continually throughout the lifecycle of the training dataset creation.

Label Studio Enterprise compares annotations from annotators and model predictions against the ground truth annotations for a task to calculate an accuracy score between 0 and 1.

Ground truth annotations are only available in Label Studio Enterprise Edition. If you’re using Label Studio Community Edition, see Label Studio Features to learn more.

Define ground truth annotations for a project

You can define ground truth annotations from a project’s Data Manager page:

  1. When viewing the data manager for a project, select the checkboxes next to annotated tasks.
  2. In the selected tasks dropdown menu, select Assign ground truths. If there are multiple annotations for a task, only the first, or earliest annotation is assigned as a ground truth.
  3. Confirm that you want to set the selected task annotations as ground truths.

You can also assign ground truths when you annotate a task.

  1. When labeling a task, create an annotation or select an existing one.
  2. Click the star icon to label the annotation as a ground truth.

Manage ground truth annotations for a project

Review and modify the ground truth annotations for a project.

Review existing ground truth annotations

You can filter the Data Manager to show only tasks with ground truth annotations so that you can review them.

Remove ground truth annotations

To remove ground truth annotations,

  1. When viewing the data manager for a project, select the checkboxes next to annotated tasks.
  2. In the selected tasks dropdown menu, select Delete ground truths. This does not delete the annotation, but changes the status of the ground truth setting for the annotation to false.

You can also remove ground truths when you annotate a task.

  1. When labeling a task, create an annotation or select an existing one.
  2. Click the star icon to label the annotation as a ground truth.