YOLO ML backend for Label Studio

The YOLO ML backend for Label Studio is designed to integrate advanced object detection, segmentation, classification, and video object tracking capabilities directly into Label Studio.

This integration allows you to leverage powerful YOLOv8 models for various machine learning tasks, making it easier to annotate large datasets and ensure high-quality predictions.

Supported Features

• LS Control Tag: Label Studio control tag from the labeling configuration.
• LS Import Supported: Indicates whether Label Studio supports Import from YOLO format to Label Studio (using the LS converter).
• LS Export Supported: Indicates whether Label Studio supports Export from Label Studio to YOLO format (the Export button on the Data Manager and using the LS converter).
• Native: Native means that only native Label Studio JSON format is supported.

Before you begin

Before you begin, you need to install the Label Studio ML backend.

This tutorial uses the YOLO example.

Quick start

1. Add LABEL_STUDIO_URL and LABEL_STUDIO_API_KEY to the docker-compose.yml file. These variables should point to your Label Studio instance and its API key, respectively. For more information about finding your Label Studio API key, see our documentation.

2. Run docker compose

   docker-compose up --build

3. Open Label Studio and create a new project with the following labeling config:

   <View>
     <Image name="image" value="$image"/>
     <RectangleLabels name="label" toName="image" model_score_threshold="0.25">
       <Label value="Car" background="blue" predicted_values="jeep,cab,limousine,truck"/>
     </RectangleLabels>
   </View>

4. Then from the Model page in the project settings, connect the model. The default URL is http://localhost:9090.

5. Add images to Label Studio.

6. Open any task in the Data Manager and see the predictions from the YOLO model.

Labeling configurations

Supported object & control tags

Object tags

• <Image> - Image to annotate
• <Video> - Video to annotate

Control tags

• <RectangleLabels> - Bounding boxes; object detection task
• <PolygonLabels> - Polygons; segmentation task
• <VideoRectangle> - Video bounding boxes; video object tracking task
• <KeyPointLabels> - Key points; pose detection task
• <Choices> - Classification

How to skip the control tag?

If you don’t want to use the ML backend for some control tags, you can force skipping by adding the model_skip="true" attribute to the control tag:

<Choices name="choice" toName="image" model_skip="true">

Mixed object and control tags

You can mix different object and control tags in one project. The YOLO model will detect all known control tags and make predictions for them. For example:

<View>
  <Image name="image1" value="$image"/>
  <RectangleLabels name="label" toName="image1" model_score_threshold="0.1">
    <Label value="person" background="red"/>
    <Label value="car" background="blue"/>
  </RectangleLabels>
  
  <Image name="image2" value="$image"/>
  <Choices name="choice" toName="image2" model_score_threshold="0.1">
    <Choice value="airship"/>
    <Choice value="passenger_car"/>
  </Choices>
</View>

In this example, both RectangleLabels and Choices will be detected and predicted by the YOLO model.

You can also use different YOLO models for the same task to compare them visually:

<View>
  <Image name="image1" value="$image"/>
  <RectangleLabels name="label1" toName="image1" model_path="yolov8n.pt" model_score_threshold="0.1">
    <Label value="car" background="blue"/>
  </RectangleLabels>
  <RectangleLabels name="label2" toName="image1" model_path="yolov8m.pt" model_score_threshold="0.1">
    <Label value="car" background="red"/>
  </RectangleLabels>
</View>

Label and choice mapping

graph TD
    A[Label Studio :: Labeling Config :: Labels or Choices] <--> B[ML Model :: Names]

If you use a common YOLO model, you have to add mapping between your labels and the ML model labels. By default, the YOLO ML backend will use the same (or lowercased) names as you specified in the value attribute.

In this example the label “Jeep” will be mapped to “jeep” in the ML model:

<Choice value="Jeep"/>

For more precise control you can use the predicted_values attribute to specify multiple and different labels from the ML model:

<Choice value="Car" predicted_values="jeep,cab,limousine"/>

**Task:**

1. **ML Model Labels:**
   - I have the following labels in my ML model:
['tench', 'goldfish', 'great_white_shark', 'tiger_shark', 'hammerhead', 'electric_ray', 'stingray', 'cock', 'hen', 'ostrich', 'brambling', 'goldfinch', 'house_finch', 'junco', 'indigo_bunting', 'robin', 'bulbul', 'jay', 'magpie', 'chickadee', 'water_ouzel', 'kite', 'bald_eagle', 'vulture', 'great_grey_owl', 'European_fire_salamander', 'common_newt', 'eft', 'spotted_salamander', 'axolotl', 'bullfrog', 'tree_frog', 'tailed_frog', 'loggerhead', 'leatherback_turtle', 'mud_turtle', 'terrapin', 'box_turtle', 'banded_gecko', 'common_iguana', 'American_chameleon', 'whiptail', 'agama', 'frilled_lizard', 'alligator_lizard', 'Gila_monster', 'green_lizard', 'African_chameleon', 'Komodo_dragon', 'African_crocodile', 'American_alligator', 'triceratops', 'thunder_snake', 'ringneck_snake', 'hognose_snake', 'green_snake', 'king_snake', 'garter_snake', 'water_snake', 'vine_snake', 'night_snake', 'boa_constrictor', 'rock_python', 'Indian_cobra', 'green_mamba', 'sea_snake', 'horned_viper', 'diamondback', 'sidewinder', 'trilobite', 'harvestman', 'scorpion', 'black_and_gold_garden_spider', 'barn_spider', 'garden_spider', 'black_widow', 'tarantula', 'wolf_spider', 'tick', 'centipede', 'black_grouse', 'ptarmigan', 'ruffed_grouse', 'prairie_chicken', 'peacock', 'quail', 'partridge', 'African_grey', 'macaw', 'sulphur-crested_cockatoo', 'lorikeet', 'coucal', 'bee_eater', 'hornbill', 'hummingbird', 'jacamar', 'toucan', 'drake', 'red-breasted_merganser', 'goose', 'black_swan', 'tusker', 'echidna', 'platypus', 'wallaby', 'koala', 'wombat', 'jellyfish', 'sea_anemone', 'brain_coral', 'flatworm', 'nematode', 'conch', 'snail', 'slug', 'sea_slug', 'chiton', 'chambered_nautilus', 'Dungeness_crab', 'rock_crab', 'fiddler_crab', 'king_crab', 'American_lobster', 'spiny_lobster', 'crayfish', 'hermit_crab', 'isopod', 'white_stork', 'black_stork', 'spoonbill', 'flamingo', 'little_blue_heron', 'American_egret', 'bittern', 'crane_(bird)', 'limpkin', 'European_gallinule', 'American_coot', 'bustard', 'ruddy_turnstone', 'red-backed_sandpiper', 'redshank', 'dowitcher', 'oystercatcher', 'pelican', 'king_penguin', 'albatross', 'grey_whale', 'killer_whale', 'dugong', 'sea_lion', 'Chihuahua', 'Japanese_spaniel', 'Maltese_dog', 'Pekinese', 'Shih-Tzu', 'Blenheim_spaniel', 'papillon', 'toy_terrier', 'Rhodesian_ridgeback', 'Afghan_hound', 'basset', 'beagle', 'bloodhound', 'bluetick', 'black-and-tan_coonhound', 'Walker_hound', 'English_foxhound', 'redbone', 'borzoi', 'Irish_wolfhound', 'Italian_greyhound', 'whippet', 'Ibizan_hound', 'Norwegian_elkhound', 'otterhound', 'Saluki', 'Scottish_deerhound', 'Weimaraner', 'Staffordshire_bullterrier', 'American_Staffordshire_terrier', 'Bedlington_terrier', 'Border_terrier', 'Kerry_blue_terrier', 'Irish_terrier', 'Norfolk_terrier', 'Norwich_terrier', 'Yorkshire_terrier', 'wire-haired_fox_terrier', 'Lakeland_terrier', 'Sealyham_terrier', 'Airedale', 'cairn', 'Australian_terrier', 'Dandie_Dinmont', 'Boston_bull', 'miniature_schnauzer', 'giant_schnauzer', 'standard_schnauzer', 'Scotch_terrier', 'Tibetan_terrier', 'silky_terrier', 'soft-coated_wheaten_terrier', 'West_Highland_white_terrier', 'Lhasa', 'flat-coated_retriever', 'curly-coated_retriever', 'golden_retriever', 'Labrador_retriever', 'Chesapeake_Bay_retriever', 'German_short-haired_pointer', 'vizsla', 'English_setter', 'Irish_setter', 'Gordon_setter', 'Brittany_spaniel', 'clumber', 'English_springer', 'Welsh_springer_spaniel', 'cocker_spaniel', 'Sussex_spaniel', 'Irish_water_spaniel', 'kuvasz', 'schipperke', 'groenendael', 'malinois', 'briard', 'kelpie', 'komondor', 'Old_English_sheepdog', 'Shetland_sheepdog', 'collie', 'Border_collie', 'Bouvier_des_Flandres', 'Rottweiler', 'German_shepherd', 'Doberman', 'miniature_pinscher', 'Greater_Swiss_Mountain_dog', 'Bernese_mountain_dog', 'Appenzeller', 'EntleBucher', 'boxer', 'bull_mastiff', 'Tibetan_mastiff', 'French_bulldog', 'Great_Dane', 'Saint_Bernard', 'Eskimo_dog', 'malamute', 'Siberian_husky', 'dalmatian', 'affenpinscher', 'basenji', 'pug', 'Leonberg', 'Newfoundland', 'Great_Pyrenees', 'Samoyed', 'Pomeranian', 'chow', 'keeshond', 'Brabancon_griffon', 'Pembroke', 'Cardigan', 'toy_poodle', 'miniature_poodle', 'standard_poodle', 'Mexican_hairless', 'timber_wolf', 'white_wolf', 'red_wolf', 'coyote', 'dingo', 'dhole', 'African_hunting_dog', 'hyena', 'red_fox', 'kit_fox', 'Arctic_fox', 'grey_fox', 'tabby', 'tiger_cat', 'Persian_cat', 'Siamese_cat', 'Egyptian_cat', 'cougar', 'lynx', 'leopard', 'snow_leopard', 'jaguar', 'lion', 'tiger', 'cheetah', 'brown_bear', 'American_black_bear', 'ice_bear', 'sloth_bear', 'mongoose', 'meerkat', 'tiger_beetle', 'ladybug', 'ground_beetle', 'long-horned_beetle', 'leaf_beetle', 'dung_beetle', 'rhinoceros_beetle', 'weevil', 'fly', 'bee', 'ant', 'grasshopper', 'cricket', 'walking_stick', 'cockroach', 'mantis', 'cicada', 'leafhopper', 'lacewing', 'dragonfly', 'damselfly', 'admiral', 'ringlet', 'monarch', 'cabbage_butterfly', 'sulphur_butterfly', 'lycaenid', 'starfish', 'sea_urchin', 'sea_cucumber', 'wood_rabbit', 'hare', 'Angora', 'hamster', 'porcupine', 'fox_squirrel', 'marmot', 'beaver', 'guinea_pig', 'sorrel', 'zebra', 'hog', 'wild_boar', 'warthog', 'hippopotamus', 'ox', 'water_buffalo', 'bison', 'ram', 'bighorn', 'ibex', 'hartebeest', 'impala', 'gazelle', 'Arabian_camel', 'llama', 'weasel', 'mink', 'polecat', 'black-footed_ferret', 'otter', 'skunk', 'badger', 'armadillo', 'three-toed_sloth', 'orangutan', 'gorilla', 'chimpanzee', 'gibbon', 'siamang', 'guenon', 'patas', 'baboon', 'macaque', 'langur', 'colobus', 'proboscis_monkey', 'marmoset', 'capuchin', 'howler_monkey', 'titi', 'spider_monkey', 'squirrel_monkey', 'Madagascar_cat', 'indri', 'Indian_elephant', 'African_elephant', 'lesser_panda', 'giant_panda', 'barracouta', 'eel', 'coho', 'rock_beauty', 'anemone_fish', 'sturgeon', 'gar', 'lionfish', 'puffer', 'abacus', 'abaya', 'academic_gown', 'accordion', 'acoustic_guitar', 'aircraft_carrier', 'airliner', 'airship', 'altar', 'ambulance', 'amphibian', 'analog_clock', 'apiary', 'apron', 'ashcan', 'assault_rifle', 'backpack', 'bakery', 'balance_beam', 'balloon', 'ballpoint', 'Band_Aid', 'banjo', 'bannister', 'barbell', 'barber_chair', 'barbershop', 'barn', 'barometer', 'barrel', 'barrow', 'baseball', 'basketball', 'bassinet', 'bassoon', 'bathing_cap', 'bath_towel', 'bathtub', 'beach_wagon', 'beacon', 'beaker', 'bearskin', 'beer_bottle', 'beer_glass', 'bell_cote', 'bib', 'bicycle-built-for-two', 'bikini', 'binder', 'binoculars', 'birdhouse', 'boathouse', 'bobsled', 'bolo_tie', 'bonnet', 'bookcase', 'bookshop', 'bottlecap', 'bow', 'bow_tie', 'brass', 'brassiere', 'breakwater', 'breastplate', 'broom', 'bucket', 'buckle', 'bulletproof_vest', 'bullet_train', 'butcher_shop', 'cab', 'caldron', 'candle', 'cannon', 'canoe', 'can_opener', 'cardigan', 'car_mirror', 'carousel', "carpenter's_kit", 'carton', 'car_wheel', 'cash_machine', 'cassette', 'cassette_player', 'castle', 'catamaran', 'CD_player', 'cello', 'cellular_telephone', 'chain', 'chainlink_fence', 'chain_mail', 'chain_saw', 'chest', 'chiffonier', 'chime', 'china_cabinet', 'Christmas_stocking', 'church', 'cinema', 'cleaver', 'cliff_dwelling', 'cloak', 'clog', 'cocktail_shaker', 'coffee_mug', 'coffeepot', 'coil', 'combination_lock', 'computer_keyboard', 'confectionery', 'container_ship', 'convertible', 'corkscrew', 'cornet', 'cowboy_boot', 'cowboy_hat', 'cradle', 'crane_(machine)', 'crash_helmet', 'crate', 'crib', 'Crock_Pot', 'croquet_ball', 'crutch', 'cuirass', 'dam', 'desk', 'desktop_computer', 'dial_telephone', 'diaper', 'digital_clock', 'digital_watch', 'dining_table', 'dishrag', 'dishwasher', 'disk_brake', 'dock', 'dogsled', 'dome', 'doormat', 'drilling_platform', 'drum', 'drumstick', 'dumbbell', 'Dutch_oven', 'electric_fan', 'electric_guitar', 'electric_locomotive', 'entertainment_center', 'envelope', 'espresso_maker', 'face_powder', 'feather_boa', 'file', 'fireboat', 'fire_engine', 'fire_screen', 'flagpole', 'flute', 'folding_chair', 'football_helmet', 'forklift', 'fountain', 'fountain_pen', 'four-poster', 'freight_car', 'French_horn', 'frying_pan', 'fur_coat', 'garbage_truck', 'gasmask', 'gas_pump', 'goblet', 'go-kart', 'golf_ball', 'golfcart', 'gondola', 'gong', 'gown', 'grand_piano', 'greenhouse', 'grille', 'grocery_store', 'guillotine', 'hair_slide', 'hair_spray', 'half_track', 'hammer', 'hamper', 'hand_blower', 'hand-held_computer', 'handkerchief', 'hard_disc', 'harmonica', 'harp', 'harvester', 'hatchet', 'holster', 'home_theater', 'honeycomb', 'hook', 'hoopskirt', 'horizontal_bar', 'horse_cart', 'hourglass', 'iPod', 'iron', "jack-o'-lantern", 'jean', 'jeep', 'jersey', 'jigsaw_puzzle', 'jinrikisha', 'joystick', 'kimono', 'knee_pad', 'knot', 'lab_coat', 'ladle', 'lampshade', 'laptop', 'lawn_mower', 'lens_cap', 'letter_opener', 'library', 'lifeboat', 'lighter', 'limousine', 'liner', 'lipstick', 'Loafer', 'lotion', 'loudspeaker', 'loupe', 'lumbermill', 'magnetic_compass', 'mailbag', 'mailbox', 'maillot_(tights)', 'maillot_(tank_suit)', 'manhole_cover', 'maraca', 'marimba', 'mask', 'matchstick', 'maypole', 'maze', 'measuring_cup', 'medicine_chest', 'megalith', 'microphone', 'microwave', 'military_uniform', 'milk_can', 'minibus', 'miniskirt', 'minivan', 'missile', 'mitten', 'mixing_bowl', 'mobile_home', 'Model_T', 'modem', 'monastery', 'monitor', 'moped', 'mortar', 'mortarboard', 'mosque', 'mosquito_net', 'motor_scooter', 'mountain_bike', 'mountain_tent', 'mouse', 'mousetrap', 'moving_van', 'muzzle', 'nail', 'neck_brace', 'necklace', 'nipple', 'notebook', 'obelisk', 'oboe', 'ocarina', 'odometer', 'oil_filter', 'organ', 'oscilloscope', 'overskirt', 'oxcart', 'oxygen_mask', 'packet', 'paddle', 'paddlewheel', 'padlock', 'paintbrush', 'pajama', 'palace', 'panpipe', 'paper_towel', 'parachute', 'parallel_bars', 'park_bench', 'parking_meter', 'passenger_car', 'patio', 'pay-phone', 'pedestal', 'pencil_box', 'pencil_sharpener', 'perfume', 'Petri_dish', 'photocopier', 'pick', 'pickelhaube', 'picket_fence', 'pickup', 'pier', 'piggy_bank', 'pill_bottle', 'pillow', 'ping-pong_ball', 'pinwheel', 'pirate', 'pitcher', 'plane', 'planetarium', 'plastic_bag', 'plate_rack', 'plow', 'plunger', 'Polaroid_camera', 'pole', 'police_van', 'poncho', 'pool_table', 'pop_bottle', 'pot', "potter's_wheel", 'power_drill', 'prayer_rug', 'printer', 'prison', 'projectile', 'projector', 'puck', 'punching_bag', 'purse', 'quill', 'quilt', 'racer', 'racket', 'radiator', 'radio', 'radio_telescope', 'rain_barrel', 'recreational_vehicle', 'reel', 'reflex_camera', 'refrigerator', 'remote_control', 'restaurant', 'revolver', 'rifle', 'rocking_chair', 'rotisserie', 'rubber_eraser', 'rugby_ball', 'rule', 'running_shoe', 'safe', 'safety_pin', 'saltshaker', 'sandal', 'sarong', 'sax', 'scabbard', 'scale', 'school_bus', 'schooner', 'scoreboard', 'screen', 'screw', 'screwdriver', 'seat_belt', 'sewing_machine', 'shield', 'shoe_shop', 'shoji', 'shopping_basket', 'shopping_cart', 'shovel', 'shower_cap', 'shower_curtain', 'ski', 'ski_mask', 'sleeping_bag', 'slide_rule', 'sliding_door', 'slot', 'snorkel', 'snowmobile', 'snowplow', 'soap_dispenser', 'soccer_ball', 'sock', 'solar_dish', 'sombrero', 'soup_bowl', 'space_bar', 'space_heater', 'space_shuttle', 'spatula', 'speedboat', 'spider_web', 'spindle', 'sports_car', 'spotlight', 'stage', 'steam_locomotive', 'steel_arch_bridge', 'steel_drum', 'stethoscope', 'stole', 'stone_wall', 'stopwatch', 'stove', 'strainer', 'streetcar', 'stretcher', 'studio_couch', 'stupa', 'submarine', 'suit', 'sundial', 'sunglass', 'sunglasses', 'sunscreen', 'suspension_bridge', 'swab', 'sweatshirt', 'swimming_trunks', 'swing', 'switch', 'syringe', 'table_lamp', 'tank', 'tape_player', 'teapot', 'teddy', 'television', 'tennis_ball', 'thatch', 'theater_curtain', 'thimble', 'thresher', 'throne', 'tile_roof', 'toaster', 'tobacco_shop', 'toilet_seat', 'torch', 'totem_pole', 'tow_truck', 'toyshop', 'tractor', 'trailer_truck', 'tray', 'trench_coat', 'tricycle', 'trimaran', 'tripod', 'triumphal_arch', 'trolleybus', 'trombone', 'tub', 'turnstile', 'typewriter_keyboard', 'umbrella', 'unicycle', 'upright', 'vacuum', 'vase', 'vault', 'velvet', 'vending_machine', 'vestment', 'viaduct', 'violin', 'volleyball', 'waffle_iron', 'wall_clock', 'wallet', 'wardrobe', 'warplane', 'washbasin', 'washer', 'water_bottle', 'water_jug', 'water_tower', 'whiskey_jug', 'whistle', 'wig', 'window_screen', 'window_shade', 'Windsor_tie', 'wine_bottle', 'wing', 'wok', 'wooden_spoon', 'wool', 'worm_fence', 'wreck', 'yawl', 'yurt', 'web_site', 'comic_book', 'crossword_puzzle', 'street_sign', 'traffic_light', 'book_jacket', 'menu', 'plate', 'guacamole', 'consomme', 'hot_pot', 'trifle', 'ice_cream', 'ice_lolly', 'French_loaf', 'bagel', 'pretzel', 'cheeseburger', 'hotdog', 'mashed_potato', 'head_cabbage', 'broccoli', 'cauliflower', 'zucchini', 'spaghetti_squash', 'acorn_squash', 'butternut_squash', 'cucumber', 'artichoke', 'bell_pepper', 'cardoon', 'mushroom', 'Granny_Smith', 'strawberry', 'orange', 'lemon', 'fig', 'pineapple', 'banana', 'jackfruit', 'custard_apple', 'pomegranate', 'hay', 'carbonara', 'chocolate_sauce', 'dough', 'meat_loaf', 'pizza', 'potpie', 'burrito', 'red_wine', 'espresso', 'cup', 'eggnog', 'alp', 'bubble', 'cliff', 'coral_reef', 'geyser', 'lakeside', 'promontory', 'sandbar', 'seashore', 'valley', 'volcano', 'ballplayer', 'groom', 'scuba_diver', 'rapeseed', 'daisy', "yellow_lady's_slipper", 'corn', 'acorn', 'hip', 'buckeye', 'coral_fungus', 'agaric', 'gyromitra', 'stinkhorn', 'earthstar', 'hen-of-the-woods', 'bolete', 'ear', 'toilet_tissue']


2. **Labeling Config:**
   - I have this labeling config from Label Studio:
   <View>
     <Image name="image" value="$image"/>
     <Choices name="choice" toName="image">
       <Choice value="Airplane"/>
       <Choice value="Car"/>
     </Choices>
   </View>

3. **Mapping Instructions:**
   - Map the labels from the Label Studio config to the closest matching ML model labels as follows:
     1. Use the `value` attribute from each `<Choice>` tag to identify the label.
     2. Find all similar and relevant labels from the ML model corresponding to each `<Choice>` label.
     3. Add a `predicted_values="<relevant_label1_from_ml_model>,<relevant_label2_from_ml_model>"` attribute inside each `<Choice>` tag using only labels from the ML model.

4. **Output:**
   - Provide the final labeling config with the `predicted_values` attribute added, using all relevant labels from the ML model, without any explanations.

YOLOv5 and other YOLO models

YOLOv8 models have been successfully tested with this ML backend.

Attempts to run YOLOv5 were unsuccessful without modifications. It may be possible to run it by applying some changes, such as installing additional dependencies. The same applies to other YOLO models.

Custom YOLO models

You can load your own YOLO labels using the following steps:

1. Mount your model as /app/models/<your-model>.pt inside of your docker.
2. Set ALLOW_CUSTOM_MODEL_PATH=true (it is true by default) in your Docker environment parameters (docker-compose.yml).
3. Add model_path="<your-model>.pt" to the control tag in the labeling configuration, e.g.:

<RectangleLabels model_path="my_model.pt">

Training

The current Label Studio ML backend doesn’t support training YOLO models. You have to do it manually on your side. Or you can contribute to this repository and add training support for this ML backend.

Classification using <Choices>

YOLO provides a classification model and Label Studio supports this with the Choices control tag.

More info: https://docs.ultralytics.com/tasks/classify/

Labeling config

<View>
  <Image name="image" value="$image"/>
  <Choices name="choice" toName="image" model_score_threshold="0.25">
    <Choice value="Airplane" predicted_values="aircraft_carrier,airliner,airship,warplane"/>
    <Choice value="Car" predicted_values="limousine,minivan,jeep,sports_car,passenger_car,police_van"/>
  </Choices>
</View>

Parameters

For example:

<Choices name="choice" toName="image" model_score_threshold="0.25" model_path="my_model.pt">

Default model

yolov8n-cls.pt is the default classification model.

Object detection using RectangleLabels

YOLO models provide bounding box detection, also known as “object detection.” Label Studio supports this with the RectangleLabels control tag.

YOLO OBB models are also supported.

More info: https://docs.ultralytics.com/tasks/detect/

Labeling config

<View>
  <Image name="image" value="$image"/>
  <RectangleLabels name="label" toName="image" model_score_threshold="0.25" opacity="0.1">
    <Label value="Person" background="red"/>
    <Label value="Car" background="blue"/>
  </RectangleLabels>
</View>

Parameters

For example:

<RectangleLabels name="label" toName="image" model_score_threshold="0.25" model_path="my_model.pt">

Default model

yolov8m.pt is the default object detection model. yolov8n-obb.pt is the default OBB object detection model.

Oriented Bounding Boxes (YOLO OBB)

Oriented (rotated) bounding boxes will be generated automatically if you use an OBB model. Specify model_obb="true" in the RectangleLabels tag to enable this mode:

<RectangleLabels name="label" toName="image" model_score_threshold="0.25" model_obb="true">

More info: https://docs.ultralytics.com/tasks/obb/

Segmentation using PolygonLabels

YOLO models provide segmentation detection, also known as “instance segmentation.” Label Studio supports this with the PolygonLabels control tag.

More info: https://docs.ultralytics.com/tasks/segment/

Labeling config

<View>
  <Image name="image" value="$image"/>
  <PolygonLabels name="label" toName="image" model_score_threshold="0.25" opacity="0.1">
    <Label value="Car" background="blue"/>
    <Label value="Person" background="red"/>
  </PolygonLabels>
</View>

Parameters

For example:

<PolygonLabels name="label" toName="image" model_score_threshold="0.25" model_path="my_model.pt">

Default model

yolov8n-seg.pt is the default segmentation model.

Keypoint detection using KeyPointLabels

YOLO models provide keypoint detection, also known as “pose estimation.” Label Studio supports this with the KeyPointLabels control tag.

More info: Ultralytics YOLO Keypoint Documentation

Labeling config

<View>
  <RectangleLabels name="keypoints_bbox" toName="image" model_skip="true">
    <Label value="person"/>
  </RectangleLabels>
  
  <KeyPointLabels name="keypoints" toName="image"
    model_score_threshold="0.75" model_point_threshold="0.5" 
    model_add_bboxes="true" model_point_size="1"
    model_path="yolov8n-pose.pt"
  >
    <Label value="nose" predicted_values="person" model_index="1" background="red" />

    <Label value="left_eye" predicted_values="person" model_index="2" background="yellow" />
    <Label value="right_eye" predicted_values="person" model_index="3" background="yellow" />

    <Label value="left_ear" predicted_values="person" model_index="4" background="purple" />
    <Label value="right_ear" predicted_values="person" model_index="5" background="purple" />
    
    <View>
      <Label value="left_shoulder" predicted_values="person" model_index="6" background="green" />
      <Label value="left_elbow" predicted_values="person" model_index="8" background="green" />
      <Label value="left_wrist" predicted_values="person" model_index="10" background="green" />

      <Label value="right_shoulder" predicted_values="person" model_index="7" background="blue" />
      <Label value="right_elbow" predicted_values="person" model_index="9" background="blue" />
      <Label value="right_wrist" predicted_values="person" model_index="11" background="blue" />
    </View>
    
    <View>
      <Label value="left_hip" predicted_values="person" model_index="12" background="brown" />
      <Label value="left_knee" predicted_values="person" model_index="14" background="brown" />
      <Label value="left_ankle" predicted_values="person" model_index="16" background="brown" />

      <Label value="right_hip" predicted_values="person" model_index="13" background="orange" />
      <Label value="right_knee" predicted_values="person" model_index="15" background="orange" />
      <Label value="right_ankle" predicted_values="person" model_index="17" background="orange" />
    </View>
  </KeyPointLabels>
  
  <Image name="image" value="$image" />
</View>

Parameters

For example:

<KeyPointLabels name="keypoints" toName="image"
                model_path="yolov8n-pose.pt"
                model_score_threshold="0.25" model_point_threshold="0.5" 
                model_add_bboxes="true" model_point_size="2">

Default model

yolov8n-pose.pt is the default keypoint detection model.

Grouping keypoints with bounding boxes

When using keypoint detection, the ML backend groups keypoints by the bounding box (bbox) associated with each detected person or object. You will see the grouping under the Regions panel on the right side of Label Studio. Note that you can drag and drop region items as necessary.

The bounding boxes are added to the prediction results by default. You can enable or disable this behavior by setting model_add_bboxes:

<KeyPointLabels name="keypoints" toName="image" model_add_bboxes="false">

<View>
  <KeyPointLabels name="keypoints" toName="image"
    model_score_threshold="0.75" model_point_threshold="0.5" 
    model_path="yolov8n-pose.pt" model_point_size="1"
    model_add_bboxes="false"              
  >
    <Label value="nose" predicted_values="person" model_index="1" background="red" />

    <Label value="left_eye" predicted_values="person" model_index="2" background="yellow" />
    <Label value="right_eye" predicted_values="person" model_index="3" background="yellow" />

    <Label value="left_ear" predicted_values="person" model_index="4" background="purple" />
    <Label value="right_ear" predicted_values="person" model_index="5" background="purple" />
    
    <View>
      <Label value="left_shoulder" predicted_values="person" model_index="6" background="green" />
      <Label value="left_elbow" predicted_values="person" model_index="8" background="green" />
      <Label value="left_wrist" predicted_values="person" model_index="10" background="green" />

      <Label value="right_shoulder" predicted_values="person" model_index="7" background="blue" />
      <Label value="right_elbow" predicted_values="person" model_index="9" background="blue" />
      <Label value="right_wrist" predicted_values="person" model_index="11" background="blue" />
    </View>
    
    <View>
      <Label value="left_hip" predicted_values="person" model_index="12" background="brown" />
      <Label value="left_knee" predicted_values="person" model_index="14" background="brown" />
      <Label value="left_ankle" predicted_values="person" model_index="16" background="brown" />

      <Label value="right_hip" predicted_values="person" model_index="13" background="orange" />
      <Label value="right_knee" predicted_values="person" model_index="15" background="orange" />
      <Label value="right_ankle" predicted_values="person" model_index="17" background="orange" />
    </View>
  </KeyPointLabels>
  <Image name="image" value="$image" />
</View>

Point mapping

For precise control, you can map keypoints to specific labels in your Label Studio configuration. Each keypoint can be associated with a specific part of a person or object, and you can define this mapping using the model_index and predicted_values attributes.

<Label value="left_eye" predicted_values="person" model_index="2" />
<Label value="right_eye" predicted_values="person" model_index="3" />

This configuration ensures that the keypoints detected by the YOLO model are correctly labeled in the Label Studio interface. For pose detection models, the model_index attribute is used to map keypoints to specific parts of the body according to the YOLO model output:

0: Nose 1: Left Eye 2: Right Eye 3: Left Ear 4: Right Ear 
5: Left Shoulder 6: Right Shoulder 7: Left Elbow 8: Right Elbow 9: Left Wrist 10: Right Wrist 
11: Left Hip 12: Right Hip 13: Left Knee 14: Right Knee 15: Left Ankle 16: Right Ankle

Recommendations

• Bounding Box Visualization: Use the model_add_bboxes parameter to visualize the bounding box containing the keypoints. This is especially useful when dealing with multiple detected persons or objects.
• Threshold Adjustment: Adjust the model_score_threshold and model_point_threshold parameters based on your dataset and the confidence level required for accurate keypoint detection.

Video object tracking using VideoRectangle

YOLO models provide object tracking, also known as “multi-object tracking.” Label Studio supports this with the VideoRectangle + Labels control tags.

More info: https://docs.ultralytics.com/modes/track/

Labeling config

<View>
    <Video name="video" value="$video"/>
    <VideoRectangle name="box" toName="video" model_tracker="botsort" model_conf="0.25" model_iou="0.7" />
    <Labels name="label" toName="video">
      <Label value="Person" background="red"/>
      <Label value="Car" background="blue"/>
    </Labels>
</View>

Trackers

https://docs.ultralytics.com/modes/track/?h=track#tracker-selection

The best tracker to use with Ultralytics YOLO depends on your specific needs.

The default tracker is BoT-SORT, which is generally well-suited for most scenarios.

However, if you’re looking for an alternative with different strengths, ByteTrack is another good choice that you can easily configure. ByteTrack is known for its high performance in multi-object tracking, especially in situations with varying object appearances and reappearances.

Both trackers can be customized using YAML configuration files to fit your specific use cases.

You can specify the tracker in the control tag:

• <VideoRectangle model_tracker="botsort">
• <VideoRectangle model_tracker="bytetrack">

Parameters for bounding boxes

The tracker works with the object detection model (bounding boxes).

The first step is to detect bounding boxes, the second step is to track them (find the same boxes among frames). These parameters are related to the first step - bounding box detection.

Read more about these parameters: https://docs.ultralytics.com/modes/track/?h=track#tracking-arguments

For example:

<VideoRectangle name="label" toName="video" model_tracker="botsort" model_conf="0.25" model_iou="0.7" />

Parameters for trackers

For an example of tracker parameters, see https://github.com/ultralytics/ultralytics/tree/main/ultralytics/cfg/trackers.

The main parameter is model_tracker which can be set to

• botsort
• bytetrack
• Or the name of any custom yaml file that you place into models directory (do not include the file extension .yaml when setting this parameter).

As long as they are available within the yaml file, you can specify tracker parameters directly from the labeling config. All parameters should be prefixed with botsort_ or bytetrack_.

For example:

<VideoRectangle 
    name="label" toName="video" 
    model_tracker="botsort" 
    botsort_track_low_thresh="0.1" botsort_track_high_thresh="0.2" 
/>

Default model

yolov8n.pt is the default object detection model.

Recommendations

• Video object tracking is a computationally intensive task. Small models like yolov8n.pt are recommended for real-time tracking, however, they may not be as accurate as larger models.

• Label Studio has timeout limits for ML backend requests. You can adjust the timeout in the Label Studio backend settings.

• Or use the CLI tool to run predictions asynchronously.

Run the YOLO ML backend

Running with Docker (recommended)

1. Start the Machine Learning backend on http://localhost:9090 with the prebuilt image:

   docker-compose up

2. Validate that the backend is running

   $ curl http://localhost:9090/
   {"status":"UP"}

3. Create a project in Label Studio. Then from the Model page in the project settings, connect the model. The default URL is http://localhost:9090.

Building from source (advanced)

To build the ML backend from source, you have to clone the repository and build the Docker image:

docker-compose build

Running without Docker (advanced)

To run the ML backend without Docker, you have to clone the repository and install all dependencies using pip:

python -m venv ml-backend
source ml-backend/bin/activate
pip install -r requirements-base.txt
pip install -r requirements.txt

Then you can start the ML backend:

label-studio-ml start ./dir_with_your_model

Also, you can check Dockerfile for additional dependencies and install them manually.

Parameters

Check the environment section in the docker-compose.yml file before running the container. All available parameters are listed there.

Note: You can use lots of YOLO model parameters in labeling configurations directly, e.g. model_path or model_score_threshold.

Command line interface for the terminal

Overview

This Command Line Interface (CLI) tool facilitates the integration of YOLO models with Label Studio for machine learning predictions. It provides an alternative method for running YOLO predictions on tasks managed by Label Studio, particularly useful for processing long videos or large datasets.

Running the model predictions directly from the CLI helps to avoid issues like connection timeouts between Label Studio and the ML backend, which can occur during lengthy processing times.

When to use the CLI

When working with extensive media files such as long videos, processing times can be significant. Label Studio may interrupt the connection with the ML backend if the request takes too long, resulting in incomplete predictions.

By running this CLI tool, you can execute model predictions asynchronously without the need for Label Studio to maintain a constant connection to the backend. This ensures that even large or complex tasks are processed fully, and predictions are saved to Label Studio using SDK once completed.

How it works

1. Label Studio Connection: The tool connects to a running instance of Label Studio using the provided API key and URL.
2. Task Preparation: Tasks can be provided directly via a JSON file or as a list of task IDs. The tool fetches task data from Label Studio if task IDs are supplied.
3. Model Loading: The YOLO model is loaded and initialized based on the project’s configuration.
4. Prediction Process: For each task, the YOLO model generates predictions, which are then post-processed to Label Studio’s expected format.
5. Asynchronous Upload: The generated predictions are uploaded back to Label Studio, allowing for large tasks to be processed without timing out.

Usage

python cli.py --ls-url http://localhost:8080 --ls-api-key your_api_key --project 1 --tasks tasks.json

or

python cli.py --ls-url http://localhost:8080 --ls-api-key YOUR_API_KEY --project 1 --tasks 1,2,3

Parameters

• --ls-url: The URL of the Label Studio instance. Defaults to http://localhost:8080.

• --ls-api-key: The API key for Label Studio. Used to authenticate the connection.

• --project: The ID of the Label Studio project where the tasks are managed. Defaults to 1.

• --tasks:

  1. The path to a JSON file containing a list of tasks or task IDs, e.g.:

  tasks_ids.json

  [1,2,3]

  tasks.json

  [{"id": 1, "data": {"image": "https://example.com/1.jpg"}}, {"id": 2, "data": {"image": "https://example.com/2.jpg"}}]

  2. If a file is not provided, you can pass a comma-separated list of task IDs directly, e.g.: 1,2,3

Logging

Use LOG_LEVEL=DEBUG to get detailed logs. Example:

LOG_LEVEL=DEBUG python cli.py --ls-url http://localhost:8080 --ls-api-key YOUR_API_KEY --project 2 --tasks 1,2,3

For developers

The architecture of the project and development guidelines are described in the README_DEVELOP.md file.