What Types of Image Segmentation Does Encord Support?

Choosing the right segmentation type shapes tool selection, quality framework, cost per label, and the kinds of questions a model can answer after training. Encord supports the three primary segmentation types. Here is what each means, how the platform handles it, and what to consider.

TL;DR

• Encord supports semantic, instance, panoptic, and 3D medical image segmentation via DICOM with MPR volume views.
• SAM (now extended to DICOM volumes) reduces annotation effort for well-bounded objects; performance degrades on specialized domains.
• Panoptic annotation requires careful ontology design and is more workflow-intensive than either semantic or instance segmentation alone.
• Label Studio's open ML backend is a better fit for teams in specialized domains that need custom segmentation models.

Semantic segmentation

Semantic segmentation assigns a class label to every pixel in an image. The output is a pixel map where each pixel belongs to one class. It answers what type of thing is here rather than which specific instance of that thing.

In Encord, semantic segmentation uses bitmask annotation with pixel-level painting tools that let annotators define class regions with precision. SAM 2 can generate initial semantic masks that annotators verify and adjust, reducing manual effort for well-bounded objects.

Common use cases: scene understanding for autonomous vehicles, medical tissue classification, and satellite image analysis for land use and vegetation type.

Instance segmentation

Instance segmentation goes further than semantic: it classifies each pixel and distinguishes individual instances of the same class. Two cars in an image become two separately labeled objects, each with their own mask, rather than a single region of car pixels.

Encord handles instance segmentation through its object annotation system. Each instance is a separately labeled object with its own polygon or bitmask. In video, object tracking maintains instance identity across frames: an object labeled in frame 1 is the same tracked instance in frame 100, even if it moves, is temporarily occluded, or changes appearance.

Instance segmentation is more expensive to annotate than semantic segmentation because each instance requires separate work. It produces training data for models that can locate and identify individual objects rather than just classify scene content.

Panoptic segmentation

Panoptic segmentation combines semantic and instance segmentation: stuff classes like background, road, and sky are labeled at the semantic level, while things classes like cars, pedestrians, and cyclists are labeled at the instance level.

Encord's ontology system can accommodate panoptic annotation schemas. Teams define separate object classes for things with instance-level annotation and classification classes for stuff with semantic annotation, then combine outputs in post-processing.

Panoptic segmentation is the most annotation-intensive approach and requires careful workflow design to ensure annotators understand and consistently apply the distinction between stuff and things labeling.

Medical imaging segmentation

Medical imaging presents unique segmentation challenges: 3D volumes in DICOM or NIfTI format, multi-plane views, and the need for clinical precision in defining anatomical boundaries.

Encord supports DICOM annotation with MPR volume support. Annotators can work across axial, sagittal, and coronal planes simultaneously. SAM has been extended to MPR DICOM volumes as of 2025, enabling AI-assisted segmentation in 3D contexts. Interpolation across slices reduces annotation burden: annotators label keyframe slices and the platform interpolates intermediate positions.

Choosing the right segmentation approach

Semantic segmentation is faster and cheaper but produces models that cannot distinguish individual instances. If the application needs to count objects or track individual items, instance or panoptic annotation is required.

Instance segmentation costs more per label but enables richer downstream tasks. The cost difference depends heavily on how many instances appear per image. Sparse scenes cost similarly to semantic segmentation. Dense scenes with many instances of the same class cost significantly more.

Panoptic annotation is most appropriate when a model genuinely needs both semantic scene understanding and individual instance identification simultaneously, typically in advanced perception systems for robotics or autonomous driving.

Label Studio's segmentation support

Label Studio supports semantic, instance, and custom segmentation schemas through its configurable template system. The Segment Anything integration is available in the open-source tier, enabling AI-assisted segmentation without requiring an enterprise contract.

The open ML backend integration lets teams connect custom or fine-tuned segmentation models directly to the annotation interface. This is particularly valuable in medical, satellite, or industrial domains where general-purpose models like SAM underperform relative to domain-specific alternatives.

You can check out our in-depth comparison of Label Studio and Encord here, or talk to an expert at HumanSignal about segmentation workflows for your CV project.