Panoptic Segmentation Updates in Encord

Panoptic Segmentation Updates in Encord

Over the past 6 months, we have updated and built new features within Encord with a strong focus on improving your panoptic segmentation workflows across data, labeling, and model evaluation. Here are some updates we’ll cover in this article:

• Bitmask lock.
• SAM + Bitmask lock + Brush for AI-assisted precision labeling.
• Fast and performant rendering of fully bitmask-segmented images and videos.
• Panoptic Quality model evaluation metrics.

Bitmask Lock within Encord Annotate to Manage Segmentation Overlap

Our Bitmask Lock feature introduces a way to prevent segmentation and masks from overlapping, providing pixel-perfect accuracy for your object segmentation tasks. By simply toggling the “Bitmask cannot be drawn over” button, you can prevent any part of a bitmask label from being included in another label. This feature is crucial for applications requiring precise object boundaries and pixel-perfect annotations, eliminating the risk of overlapping segmentations.

Let’s see how to do this within Encord Annotate:

Step 1: Create your first Bitmask

Initiating your labeling process with the Bitmask is essential for creating precise object boundaries. If you are new to the Bitmask option, check out our quickstart video walkthrough on creating your first Bitmask using brush tools for labeling.

Step 2: Set Bitmask Overlapping Behavior 

Managing how bitmasks overlap is vital for ensuring accurate segmentation, especially when dealing with multiple objects that are close to each other or overlapping.

After creating your first bitmask, adjust the overlapping behavior settings to dictate how subsequent bitmasks interact with existing ones. This feature is crucial for delineating separate objects without merging their labels—perfect for panoptic segmentation.

This prevents any part of this bitmask label from being included in another label. This is invaluable for creating high-quality datasets for training panoptic segmentation models.

Step 3: Lock Bitmasks When Labeling Multiple Instances

Different images require different approaches. Beyond HSV, you can use intensity values for grayscale images (like DICOM) or RGB for color-specific labeling. This flexibility allows for tailored labeling strategies that match the unique attributes of your dataset.

Experiment with the different settings (HSV, intensity, and RGB) to select the best approach for your specific labeling task. Adjust the criteria to capture the elements you need precisely.

Step 4: Using the Eraser Tool

Even with careful labeling, adjustments may be necessary. The eraser tool can remove unwanted parts of a bitmask label before finalizing it, providing an extra layer of precision.

If you've applied a label inaccurately, use the eraser tool to correct any errors by removing unwanted areas of the bitmask.

Bitmask-Segmented Images and Videos Got a Serious Performance Lift (At Least 5x)

Encord's commitment to enhancing user experience and efficiency is evident in the significant performance improvements made to the Bitmask-segmented annotation within the Label Editor. Our Engineering team has achieved a performance lift of at least 5x by directly addressing user feedback and pinpointing critical bottlenecks. This improves how fast the editor loads for your panoptic segmentation labeling instances. 

Here's a closer look at the differences between the "before" and "after" scenarios, highlighting the advancements:

Before the Performance Improvements:

• Performance Lag on Zoom: Users experienced small delays when attempting to zoom in on images, with many instances (over 100) that impacted the precision and speed of their labeling process.
• Slow Response to Commands: Basic functionalities like deselecting tools or simply navigating through the label editor were met with sluggish responses.
• Operational Delays: Every action, from image loading to applying labels, was hindered by "a few milliseconds" of delay, which accumulated significant time overheads across projects.

After the Performance Enhancements:

• Quicker Image Load Time: The initial step of image loading has seen a noticeable speed increase! This sets a good pace for the entire labeling task.
• Responsiveness: The entire label editor interface, from navigating between tasks to adjusting image views, is now remarkably more responsive. This change eradicates previous lag-related frustrations and allows for a smoother user experience.
• Improved Zoom Functionality: Zooming in and out has become significantly more fluid and precise. This improvement is precious for detailed labeling work, where accuracy is paramount.

The positive changes directly result from the Engineering team's responsiveness to user feedback. Our users have renewed confidence in handling future projects with the Label Editor. We are dedicated to improving Encord based on actual user experiences.

Use Segment Anything Model (SAM) and Bitmask Lock for High Annotation Precision

Starting your annotation process can be time-consuming, especially for complex images. Our Segment Anything Model (SAM) integration offers a one-click solution to create initial annotations. SAM identifies and segments objects in your image, significantly speeding up the annotation process while ensuring high accuracy.

Step 1: Select the SAM tool from the toolbar with the Bitmask Lock enabled. 

Step 2: Click on the object you wish to segment in your image. SAM will automatically generate a precise bitmask for the object.

Step 3: Use the bitmask brush to refine the edges for pixel-perfect segmentation if needed.

Validate Segmentation with Panoptic Quality Metrics

You can easily evaluate your segmentation model’s panoptic mask quality with new metrics: 

• mSQ (mean Segmentation Quality)
• mRQ (mean Recognition Quality)
• mPQ (mean Panoptic Quality)

The platform will calculate mSQ, mRQ, and mPQ for your predictions, labels, and dataset to clearly understand the segmentation performance and areas for improvement.

Navigate to Active → Under the Model Evaluation tab, choose the panoptic model you want to evaluate. Under Display, toggle the Panoptic Quality Metrics (still in beta) option to see the model's mSQ, mRQ, and mPQ scores.

Fast Rendering of Fully Bitmask-Segmented Images within Encord Active

The performance improvement within the Label Editor also translates to how you view and load panoptic segmentation within Active. 

Try it yourself:

Evaluate your models and build active learning pipelines with Encord

Try Encord today

Key Takeaways: Panoptic Segmentation Updates in Encord

Here’s a recap of the key features and improvements within Encord that can improve your Panoptic Segmentation workflows across data and models:

• Bitmask Lock: This feature prevents overlaps in segmentation. it guarantees the integrity of each label, enhancing the quality of the training data and, consequently, the accuracy of machine learning models. This feature is crucial for projects requiring meticulous detail and precision.
• SAM + Bitmask Lock + Brush: The Lock feature allows you to apply Bitmasks to various objects within an image, which reduces manual effort and significantly speeds up your annotation process. The integration of SAM within Encord's platform, using Lock to manage Bitmask overlaps, and the generic brush tool empower you to achieve precise, pixel-perfect labels with minimal effort.
• Fast and Performant Rendering of Fully Bitmask-segmented Images and Videos: We have made at least 5x improvements to how Encord quickly renders fully Bitmask-segmented images and videos across Annotate Label Editor and Active.
• Panoptic Quality Model Evaluation Metrics: The Panoptic Quality Metrics—comprising mean Segmentation Quality (mSQ), mean Recognition Quality (mRQ), and mean Panoptic Quality (mPQ)—provide a comprehensive framework for evaluating the effectiveness of segmentation models.