How to use SAM to Automate Data Labeling in Encord

Meta recently released their new Visual Foundation Model (VFM) called the Segment Anything Model (SAM). An open-source VFM with powerful auto-segmentation workflows. Here’s our guide for how to use SAM to automate data labeling in Encord. 

As data ops, machine learning (ML), and annotation teams know, labeling training data from scratch is time intensive and often requires expert labeling teams and review stages. Manual data labeling can quickly become expensive, especially for teams still developing best practices or annotating large amounts of data.

Efficiently speeding up the data labeling process can be a challenge; this is where automation comes in.

Incorporating automation into your workflow is one of the most effective ways to produce high-quality data fast.

Last week Meta released their new Visual Foundation Model (VFM) called the Segment Anything Model (SAM), an open-source VFM with incredible abilities to support auto-segmentation workflows.

Given this new release, Encord is excited to introduce the first SAM-powered auto-segmentation tool to help teams generate high-quality segmentation masks in seconds.

Create quality masks with our SAM-powered auto-segmentation tool

Whether you're new to labeling data for your model or have several models in production, our new SAM-powered auto-segmentation tool can help you save time and streamline your labeling process. To maximize the benefits of this tool, follow these steps:

1. Set up your image segmentation project by attaching your dataset and ontology. 

2. Next, access our image auto-segmentation tool from the left-hand side toolbar.

Steps 1 & 2 for using SAM with Encord

3. Prompt the SAM model with a key point or bounding box (Grid prompting coming soon, let us know if you want to help us test it out!) on or over the area you wish to segment. To speed up your workflow, you can draw multiple bounding boxes simultaneously, and our SAM-powered auto-segmentation tool will label them for you.

Step 3 for using SAM with Encord

4. In a split second, our tool will calculate the image embeddings in the image encoder and generate a pixel-perfect segmentation mask for you using the prompt and prompt encoder.

5.  You can use our drawing/eraser tool to add to or subtract from the mask to adjust it.  If you accidentally create a mask over an area you don’t like, use the erase tool to erase it.

Steps 4 & 5 for using SAM with Encord

Integrate an AI-assisted Micro-model to make labeling even faster 

While AI-assisted tools such as SAM-powered auto-segmentation can be great for teams just getting started with data labeling, teams who follow more advanced techniques like pre-labeling can take automation to the next level with micro-models.

By using your model or a Micro-model, pre-labels can significantly increase labeling efficiency. As the model improves with each iteration, labeling costs decrease, allowing teams to focus their manual labeling efforts on edge cases or areas where the model may not perform as well. This results in faster, less expensive labeling with improved model performance.

Check out our case study to learn more about our pre-labeling workflow, powered by AI-assisted labeling, and how one of our customers increased their labeling efficiency by 37x using AI-assisted Micro-models.

Try our auto-segmentation tool on an image labeling project or start using model-assisted labeling today.

Make foundation models your own.

Label with SAM in Encord

Key Takeaways

Meta's Segment Anything Model (SAM) is a powerful and effective open-source Visual Foundation Model (VFM) that will make a positive difference to automated segmentation workflows for computer vision and ML projects.

AI-assisted labeling can reduce labeling costs and improve with each iteration. Our SAM-powered auto-segmentation tool and AI-assisted labeling workflow are available to all customers. We're excited for our users to see how automation can significantly reduce costs and increase labeling efficiency.

Ready to improve the performance and scale your data operations, labeling, and automated annotation? 

Sign-up for an Encord Free Trial: The Active Learning Platform for Computer Vision, used by the world’s leading computer vision teams. 

AI-assisted labeling, model training & diagnostics, find & fix dataset errors and biases, all in one collaborative active learning platform, to get to production AI faster. Try Encord for Free Today. 

Want to stay updated?

• Follow us on Twitter and LinkedIn for more content on computer vision, training data, and active learning.
• Join the Slack community to chat and connect.

Segment Anything Model (SAM) in Encord Frequently Asked Questions (FAQs)

What is a segmentation mask?

A segmentation mask assigns a label to each pixel in an image, identifying objects or regions of interest. This is done by creating a binary image where object pixels are marked with 1, and the rest with 0.  It's used in computer vision for object detection and image segmentation and for training machine learning models for image recognition.

How does the SAM-powered auto-segmentation work?

Combining SAM with Encord Annotate offers users an auto-segmentation tool with powerful ontologies, an interactive editor, and media support. SAM can segment objects or images without prior exposure using basic keypoint info and a bounding box. Encord annotate utilizes SAM to annotate various media types such as images, videos, satellite data, and DICOM data.

If you want to better understand SAM’s inner workings and importance, please refer to the SAM explainer.

Can I use SAM in Encord for bounding boxes?

Encord’s auto-segmentation feature supports various types of annotations such as bounding boxes, polyline annotation, and keypoint annotation. Additionally, Encord utilizes SAM for annotating images, videos, and specializes data types including satellite (SAR), DICOM, NIfTI, CT, X-ray, PET, ultrasound and MRI files. 

For more information on auto-segmentation for medical imaging computer vision models, please refer to the documentation.

Can I Fine-tune SAM?

The image encoder of SAM is a highly intricate architecture containing numerous parameters. To fine-tune the model, it is more practical to concentrate on the mask encoder instead, which is lightweight and, therefore simpler, quicker, and more efficient in terms of memory usage. Please read Encord’s tutorial on how to fine-tune Segment Anything.

Can I try SAM in Encord for free?

Encord has integrated its powerful ontologies, interactive editor, and versatile data type support with SAM to enable segmentation of any type of data. SAM's auto-annotation capability can be utilized for this purpose. Encord offers a free trial that can be accessed by logging in to our platform or please contact our team to get yourself started 😀