How to build a labeling tool for medical image multi modality review CT MRI X-ray
When clinicians need to evaluate complex radiological cases, they expect specialized tools.
Building a custom interface for medical image multi-modality review (CT / MRI / X-ray) requires balancing workflow ergonomics with strict data privacy constraints.
With Label Studio, you can deploy a specialized annotation environment for this exact task without writing a frontend application from scratch.
Deploy an interface from a plain-language specification using a coding agent.
Display side-by-side radiological studies alongside paginated volumetric slices.
Capture localized findings with polygons and global diagnostic classifications.
Collect pairwise preference data to compare imaging quality across modalities.
Keep protected health information secure within self-hosted deployment boundaries.
The problem
Labeling for medical image multi-modality review (CT / MRI / X-ray) introduces high friction because reviewers must correlate findings across disparate data formats like static 2D radiographs and 3D slice stacks.
Clinical subject matter experts lose valuable time context-switching between basic tools or struggling with interfaces that lack synchronized comparison layouts. Handling this data also triggers severe compliance obligations under regulations like HIPAA, meaning you cannot push protected patient images to untrusted public cloud platforms. Building a compliant and performant viewer with region-drawing capabilities from scratch costs engineering teams months of development time before a single medical image gets labeled.
The short answer
Label Studio serves as the foundation for your annotation tool, while a coding agent writes the exact user interface configuration you need. The agent combines the XML labeling config builder skill to translate a plain-language specification into an optimized layout and the Label Studio SDK/CLI to wire that configuration into a live project programmatically. Rather than building a new labeling application from scratch, agents generate the interface from your spec and deploy it into Label Studio in one pass.
Docs: Label Studio tags → https://labelstud.io/tags/image
Docs: Multi-page document templates → https://labelstud.io/templates/multi-page-document-annotation
Docs: Importing pre-annotations → https://labelstud.io/guide/predictions
LLM-friendly docs (markdown) → https://labelstud.io/llms.txt
What you're building
View a static 2D chest X-ray side-by-side with a volumetric CT series in a unified workspace.
Navigate through multi-slice MRI or CT stacks using built-in pagination controls.
Draw precise polygons around lung nodules or lesions directly on the relevant anatomical plane.
Assign global diagnostic classifications to the entire multi-modality study using single-choice pickers.
Compare imaging quality between two distinct modalities using a pairwise preference control.
Enter free-text clinical rationale for complex diagnostic decisions using a dedicated notes field.
How to build it in Label Studio
1. Set up the project
Deploy a self-hosted instance of Label Studio to keep protected health information within your compliant infrastructure boundary. Format a single labeling unit for medical image multi-modality review (CT / MRI / X-ray) as a JSON object containing a pre-converted 2D image URL for the X-ray and an array of slice URLs for the volumetric study. Include metadata fields for the patient identifier, the study identifiers, and the specific modalities present so reviewers can filter their queues effectively. Apply standard de-identification profiles like the DICOM Security and System Management Profiles before upload to ensure no protected data leaks into the annotation environment. Load any required clinical ontology files into your environment early so annotators select from standardized diagnostic codes.
2. Generate the labeling interface with the XML config skill
Pass your interface requirements from the previous section to a coding agent equipped with the XML labeling config builder skill. The agent processes your plain-language description and emits a validated Label Studio XML configuration tailored for medical image multi-modality review (CT / MRI / X-ray). This generated markup applies the exact layout structures and specific control tags required to map clinical decisions back to your source data.
<Image name="..." value="..." zoom="true"> https://labelstud.io/tags/image — displays the static radiograph or the paginated multi-slice stack for visual review during medical image multi-modality review (CT / MRI / X-ray).
<PolygonLabels name="..." toName="..." ...> https://labelstud.io/tags/polygonlabels.html — enables clinicians to draw precise boundaries around targeted anatomical structures or pathological findings.
<RectangleLabels name="..." toName="..." ...> https://labelstud.io/tags/rectanglelabels.html — provides a bounding box tool to quickly localize general regions of interest across the scan.
<Pairwise name="..." toName="..." ...> https://labelstud.io/tags/pairwise.html — captures direct preference decisions when comparing the diagnostic clarity of two different imaging studies.
3. Wire it into a project with the SDK
Instruct the agent to run the Label Studio SDK/CLI to create a new project and inject the generated XML configuration. The agent can then upload your prepared JSON tasks and insert any machine learning model predictions as static pre-annotations to give human reviewers a head start, which is highly applicable for medical image multi-modality review (CT / MRI / X-ray). Use this same agent loop to iterate on the configuration rapidly. Run a small batch of cases, watch clinical annotators interact with the layout, ask the agent to regenerate the XML to fix usability issues, and redeploy the updated interface.
4. Set up review and quality workflows
Configure a multi-annotator overlap percentage greater than one to ensure multiple radiologists evaluate ambiguous or complex cases. Route the resulting submissions into a dedicated reviewer queue where a senior subject matter expert can resolve disagreements and finalize the ground truth. Track specific agreement metrics that matter for medical image multi-modality review (CT / MRI / X-ray), such as bounding box intersection over union (IoU) for lesion localization and classification agreement for the overall diagnostic decision. If the IoU falls below your target threshold, refine the labeling instructions and retrain the annotation team.
5. Export and integrate
Export your finalized annotations in the default JSON format to capture the full hierarchical structure of the multi-modality data. Downstream consumers of medical image multi-modality review (CT / MRI / X-ray) will extract the coordinate arrays for drawn regions, the specific slice index where a finding occurs, and the selected diagnostic choices. Pass this structured payload into your computer vision training pipeline, load the clinical decisions into an analytics warehouse, or feed the results into an evaluation harness to audit model performance in production.
Why Label Studio for medical image multi-modality review (CT / MRI / X-ray)
Deploy a self-hosted architecture to satisfy strict HIPAA compliance rules and prevent protected patient data from leaving your secure network.
Combine the multi-page pagination template with side-by-side flex layouts to eliminate the friction of context-switching between separate viewers.
Import static model predictions directly into the workspace to accelerate the slow process of manually localizing every small lung nodule from scratch.
Define custom keyboard shortcuts for bounding boxes and polygons to reduce the ergonomic strain of annotating high volumes of volumetric slices.
Implement native reviewer queues and agreement metrics to systematically resolve diagnostic disagreements between multiple specialized clinicians.
Common variations
Segmenting specific anatomical structures in prenatal ultrasound sweeps using polygon tools and video timeline scrubbing.
Classifying whole-slide pathology images by mapping high-resolution tissue patches in a multi-image grid layout.
Tracking tumor progression over time by comparing longitudinal PET scan slices using pairwise preference controls.
Validating automated fracture detection models on orthopedic radiographs by correcting pre-annotated bounding boxes.
Next steps
XML labeling config builder skill → https://github.com/HumanSignal/create-xml-labeling-config-skill
Label Studio SDK/CLI → https://api.labelstud.io/api-reference/introduction/getting-started
LLM-friendly docs (markdown) → https://labelstud.io/llms.txt
Multi-page document templates → https://labelstud.io/templates/multi-page-document-annotation
Importing pre-annotations → https://labelstud.io/guide/predictions
How do you de-identify medical images before uploading them to the labeler?
Strip protected health information from image headers and burned-in pixels before the data reaches the annotation environment. Apply the DICOM PS3.15 Security and System Management Profiles to standardize the scrubbing process. If you fall under HIPAA regulations, ensure your pipeline meets the Safe Harbor standard by removing all 18 identifiers before converting the files to web-friendly images.
How do you configure the interface to handle volumetric scans like CT and MRI?
Convert volumetric slices into standard web image formats and pass them as an array of URLs to your task payload. Configure the Image tag with the valueList property instead of a single value to generate a paginated stack. This setup allows reviewers to scroll through axial or coronal slices directly in the workspace while comparing them to static radiographs in a side-by-side layout.
How do you manage token expiration when pulling data from public imaging archives?
Public repositories like The Cancer Imaging Archive (TCIA) use OAuth tokens that expire over time. Design your ingestion pipeline to cache pulled studies locally and implement automated token refresh logic before creating annotation tasks. Rely on immutable local storage for the converted web images so active human review sessions do not crash when external API tokens timeout.
How do you format machine learning predictions for medical image pre-annotations?
Format your pre-annotations as a predictions array within the task JSON. The prediction objects must exactly match your target control tags, such as PolygonLabels for precise lesion boundaries or RectangleLabels for bounding boxes. Include the model version and confidence score in the payload so downstream evaluation pipelines can track active learning improvements over time.
What metrics should you track for inter-annotator agreement on localized findings?
Track the intersection over union metric to measure how closely different radiologists agree on the exact boundaries of a drawn polygon or bounding box. For overall diagnostic decisions, track Fleiss' kappa to evaluate categorical agreement across multiple clinicians. Route cases that fall below your target intersection over union threshold to a senior subject matter expert queue for final resolution.