NEW OpenAI Structured Outputs with Label Studio 🚀

Label Studio Task Format

Label Studio JSON format of annotated tasks

When you annotate data, Label Studio stores the output in JSON format. The raw JSON structure of each completed task uses the following example: 

{
    "id": 1,
    "created_at":"2021-03-09T21:52:49.513742Z",
    "updated_at":"2021-03-09T22:16:08.746926Z",
    "project":83,

    "data": {
        "image": "https://example.com/opensource/label-studio/1.jpg"
    },

    "annotations": [
        {
            "id": "1001",
            "result": [
                {
                    "from_name": "tag",
                    "id": "Dx_aB91ISN",
                    "source": "$image",
                    "to_name": "img",
                    "type": "rectanglelabels",
                    "value": {
                        "height": 10.458911419423693,
                        "rectanglelabels": [
                            "Moonwalker"
                        ],
                        "rotation": 0,
                        "width": 12.4,
                        "x": 50.8,
                        "y": 5.869797225186766
                    }
                }
            ],
            "was_cancelled":false,
            "ground_truth":false,
            "created_at":"2021-03-09T22:16:08.728353Z",
            "updated_at":"2021-03-09T22:16:08.728378Z",
            "lead_time":4.288,
            "result_count":0,
            "task":1,
            "completed_by":10
        }
    ],

    "predictions": [
        {
            "created_ago": "3 hours",
            "model_version": "model 1",
            "result": [
                {
                    "from_name": "tag",
                    "id": "t5sp3TyXPo",
                    "source": "$image",
                    "to_name": "img",
                    "type": "rectanglelabels",
                    "value": {
                        "height": 11.612284069097889,
                        "rectanglelabels": [
                            "Moonwalker"
                        ],
                        "rotation": 0,
                        "width": 39.6,
                        "x": 13.2,
                        "y": 34.702495201535505
                    }
                }
            ]
        },
        {
            "created_ago": "4 hours",
            "model_version": "model 2",
            "result": [
                {
                    "from_name": "tag",
                    "id": "t5sp3TyXPo",
                    "source": "$image",
                    "to_name": "img",
                    "type": "rectanglelabels",
                    "value": {
                        "height": 33.61228406909789,
                        "rectanglelabels": [
                            "Moonwalker"
                        ],
                        "rotation": 0,
                        "width": 39.6,
                        "x": 13.2,
                        "y": 54.702495201535505
                    }
                }
            ]
        }
    ]
}

Relevant JSON property descriptions

Review the full list of JSON properties in the API documentation.

JSON property name Description
id Identifier for the labeling task from the dataset.
data Data copied from the input data task format. See the documentation for Task Format.
project Identifier for a specific project in Label Studio.
annotations Array containing the labeling results for the task.
annotations.id Identifier for the completed task.
annotations.lead_time Time in seconds to label the task.
annotations.result Array containing the results of the labeling or annotation task.
annotations.updated_at Timestamp for when the annotation is created or modified.
annotations.completed_at Timestamp for when the annotation is created or submitted.
annotations.completed_by User ID of the user that created the annotation. Matches the list order of users on the People page on the Label Studio UI.
annotations.was_cancelled Boolean. Details about whether or not the annotation was skipped, or cancelled.
result.id Identifier for the specific annotation result for this task.
result.from_name Name of the tag used to label the region. See control tags.
result.to_name Name of the object tag that provided the region to be labeled. See object tags.
result.type Type of tag used to annotate the task.
result.value Tag-specific value that includes details of the result of labeling the task. The value structure depends on the tag for the label. For more information, see Explore each tag.
drafts Array of draft annotations. Follows similar format as the annotations array. Included only for tasks exported as a snapshot from the UI or using the API.
predictions Array of machine learning predictions. Follows the same format as the annotations array, with one additional parameter.
predictions.score The overall score of the result, based on the probabilistic output, confidence level, or other.
task.updated_at Timestamp for when the task or any of its annotations or reviews are created, updated, or deleted.

How Label Studio saves results in annotations

Each annotation result consists of list or items, also called regions, stored under annotation.result field:

{"result": [{"id": "123", ...}, {"id": "456", ...}], ...}

Regions can represent any annotation action - drawn bounding box, created relation, assigned category etc. Each "id" field formed as a string with the characters A-Za-z0-9_- and assigned according to the structure of annotation: each individual labeling entity gets its own ID, and same IDs are used to link different entities together - for example, relate them with <Relation>, or assign conditional annotation by using "perRegion" attribute.

The format is inferred by the following principles:

  • there is always should be at least 1 object tag
  • object tags define data types used (image, html, video, etc.)
  • at least 1 control tag should be attached to object tag to create regions on that object
  • every control tag produces 1 result for region
  • results for the same region share the same id
  • classification technically creates special empty classification region

When a prediction is used to create an annotation, the result IDs stay the same in the annotation field. This allows you to track the regions generated by your machine learning model and compare them directly to the human-created and reviewed annotations.

The "value" signifies the outcome of the labeling process, and its structure varies based on the labeling configuration. To understand the structure for specific tags, please refer to the Control tags documentation and explore the relevant tag.

Examples

per-regions

Example of result format when conditional "perRegion" structure is enforced:

<Image name="image" value="$image"/>
<RectangleLabels name="product" toName="image">
  <Label value="Some label" />
  ...
</RectangleLabels>
<TextArea name="name" toName="image" perRegion="true" />

1–2 results for one region: required labeling and optional per-region text result

[{
  "id": "X_12fGk",
  "from_name": "product",
  "to_name": "image",
  "type": "rectanglelabels",
  // ...
  "value": {
    "labels": ["Some label"],
    // ...
  }
}, {
  "id": "X_12fGk",
  "from_name": "name",
  "to_name": "image",
  "type": "textarea",
  // ...
  "value": {
    "text": ["Roasted beans"],
    // ...
  }
}]

optional labels

<Image name="image" value="$image"/>
<Rectangle name="product" toName="image" />
<Labels name="kind" toName="image">
  <Label value="Tea" />
  <Label value="Coffee" />
</Labels>

1–2 results for one region: required drawing tool and optional labeling attached to it

[{
  "id": "X_12fGk",
  "from_name": "product",
  "to_name": "image",
  "type": "rectangle",
  // ...
  "value": {
    "x": 100,
    "y": 200,
    // ...
  }
}, {
  "id": "X_12fGk",
  "from_name": "kind",
  "to_name": "image",
  "type": "labels",
  // ...
  "value": {
    "labels": ["Tea"],
    // ...
  }
}]

multi-labels

<Image name="image" value="$image"/>
<Rectangle name="product" toName="image" />
<Labels name="kind" toName="image">
  <Label value="Tea" />
  <Label value="Coffee" />
</Labels>
<Labels name="country" toName="image">
  <Label value="Sri-Lanka" />
  <Label value="Brazil" />
</Labels>
<Number name="price" toName="image" perRegion="true" required="true" />

1-4 results for one region: optional labels and required price + rectangle itself

[{
  "id": "X_12fGk",
  "from_name": "product",
  "to_name": "image",
  "type": "rectangle",
  // ...
  "value": {
    // rectangle sizes
  }
}, {
  "id": "X_12fGk",
  "from_name": "kind",
  "to_name": "image",
  "type": "labels",
  // ...
  "value": {
    "labels": ["Coffee"],
    // rectangle sizes
  }
}, {
  "id": "X_12fGk",
  "from_name": "country",
  "to_name": "image",
  "type": "labels",
  // ...
  "value": {
    "labels": ["Brazil"],
    // rectangle sizes
  }
}, {
  "id": "X_12fGk",
  "from_name": "price",
  "to_name": "image",
  "type": "labels",
  // ...
  "value": {
    "number": 12.5,
    // rectangle sizes
  }
}]

relations

You can draw relation arrow between two objects. For example, object detection configuration

<Image name="image" value="$image"/>
<RectangleLabels name="kind" toName="image">
  <Label value="Car" />
  <Label value="Airplaine" />
</RectangleLabels>
```json
[{
  "id": "oid67",
  "type": "rectanglelabels",
  // ...
},
{
  "id": "RQbW3Sj_Zr",
  "type": "rectanglelabels",
  // ...
},
{
  "type": "relation",
  "to_id": "RQbW3Sj_Zr",
  "from_id": "oid66",
  "direction": "right"
}]

In this article

  1. Label Studio JSON format of annotated tasks
    1. Relevant JSON property descriptions
  2. How Label Studio saves results in annotations
  3. Examples
    1. per-regions
    2. optional labels
    3. multi-labels
    4. relations
Unlock more with Enterprise Get a Demo