If you thought the AI space was already moving fast with ChatGPT, GPT4, and Stable Diffusion, then strap in and get ready for the next groundbreaking innovation in AI. Meta’s FAIR lab has just released the Segment Anything Model (SAM), a state-of-the-art image segmentation model that aims to change the field of computer vision.  SAM is based on foundation models that have had a significant impact on natural language processing (NLP). It focuses on promptable segmentation tasks, using prompt engineering to adapt to diverse downstream segmentation problems. In this blog post, you will:  Understand what SAM is and why it’s a game-changer. Learn how it compares to previous models. Dive into SAM's network architecture, design, and implementation. Discover potential uses of SAM for AI-assisted labeling. {{light_callout_start}} Update: Segment Anything (SAM) is live in Encord! Check out our tutorial on How To Fine-Tune Segment Anything or book a demo with the team to learn how to use the Segment Anything Model (SAM) to reduce labeling costs. {{light_callout_end}} Why are we so excited about SAM?  Having tested it out for a day now, we can see the following incredible advances: SAM can segment objects by simply clicking or interactively selecting points to include or exclude from the object. You can also create segmentations by drawing bounding boxes or segmenting regions with a polygon tool and it will snap to the object. When encountering uncertainty in identifying the object to be segmented, SAM is capable of producing multiple valid masks. SAM has the ability to identify and generate masks for all objects present in an image automatically. After precomputing the image embeddings, SAM can provide a segmentation mask for any prompt instantly, enabling real-time interaction with the model. {{SAM_CTA}} A Brief History of Meta's AI & Computer Vision As one of the leading companies in the field of artificial intelligence (AI), Meta has been pushing the boundaries of what's possible with machine learning models: from recently released open source models such as LLaMA to developing the most used Python library for ML and AI, PyTorch. Advances in Computer Vision Computer vision has also experienced considerable advancements, with models like CLIP bridging the gap between text and image understanding. These models use contrastive learning to map text and image data. This allows them to generalize to new visual concepts and data distributions through prompt engineering.  FAIR’s Segment Anything Model (SAM) is the latest breakthrough in this field. Their goal was to create a foundation model for image segmentation that can adapt to various downstream tasks using prompt engineering. Let’s shortly explore some of the key developments in computer vision that have contributed to the growth of AI systems like Meta's. Convolutional Neural Networks (CNNs) CNNs, first introduced by Yann LeCun (now VP & Chief AI scientist at Meta) in 1989, have emerged as the backbone of modern computer vision systems, enabling machines to automatically learn and recognize complex patterns in images.  By employing convolutional layers, CNNs can capture local and global features in images, allowing them to effectively recognize objects, scenes, and actions. This has led to significant improvements in tasks such as image classification, object detection, and semantic segmentation. {{light_callout_start}} Interested in learning more about CNNs? Read our Convolutional Neural Networks (CNN) Overview. {{light_callout_end}} Generative Adversarial Networks (GANs) GANs are a type of deep learning model that Ian Goodfellow and his team came up with in 2014. They are made up of two neural networks, a generator and a discriminator, that compete with each other. The generator aims to create realistic outputs, while the discriminator tries to distinguish between real and generated outputs. The competition between these networks has resulted in the creation of increasingly realistic synthetic images and has led to advances in tasks such as image synthesis, data augmentation, and style transfer. Transfer Learning and Pre-trained Models Similar to NLP, computer vision has benefited from the development of pre-trained models that can be fine-tuned for specific tasks. Models such as ResNet, VGG, and EfficientNet have been trained on large-scale image datasets, allowing researchers to use these models as a starting point for their own projects. The Growth of Foundation Models Foundation models in natural language processing (NLP) have made significant strides in recent years, with models like Meta’s own LLaMa or OpenAI’s GPT-4 demonstrating remarkable capabilities in zero-shot and few-shot learning.  These models are pre-trained on vast amounts of data and have the ability to generalize to new tasks and data distributions by using prompt engineering. Meta AI has been instrumental in advancing this field, fostering research and the development of large-scale NLP models that have a wide range of applications. Here, we explore the factors contributing to the growth of foundation models. Large-scale Language Models The advent of large-scale language models like GPT-4 has been a driving force behind the development of foundation models in NLP. These models employ deep learning architectures with billions of parameters, allowing them to capture complex patterns and structures in the training data.  Transfer Learning A key feature of foundation models in NLP is their capacity for transfer learning. Once trained on a large corpus of data, they can be fine-tuned on smaller, task-specific datasets to achieve state-of-the-art performance across a variety of tasks. Zero-shot and Few-shot Learning Foundation models have also shown promise in zero-shot and few-shot learning, where they can perform tasks without any fine-tuning or with minimal task-specific training data. This capability is largely attributed to the models' ability to understand and generate human-like responses based on the context provided by prompts. {{try_encord}} Multi-modal Learning Another growing area of interest is multi-modal learning, where foundation models are trained to understand and generate content across different modalities, such as text and images.  Models like CLIP and ALIGN show how NLP and computer vision could be used together to make multi-modal models that can translate actions from one domain to another. Ethical Considerations and Safety The growth of foundation models in NLP has also raised concerns about their ethical implications and safety. Researchers are actively exploring ways to mitigate potential biases, address content generation concerns, and develop safe and controllable AI systems. Proof of this was the recent call for a six-month halt on all development of cutting edge models. Comparing Segment Anything Model to Previous Models SAM is a big step forward for AI because it builds on the foundations that were set by earlier models. SAM can take input prompts from other systems, such as, in the future, taking a user's gaze from an AR/VR headset to select an object, using the output masks for video editing, abstracting 2D objects into 3D models, and even popular Google Photos tasks like creating collages. It can handle tricky situations by generating multiple valid masks where the prompt is unclear. Take, for instance, a user’s prompt for finding Waldo: Source One of the reasons the results from SAM are groundbreaking is because of how good the segmentation masks are compared to other techniques like ViTDet. The illustration below shows a comparison of both techniques:  Source {{light_callout_start}} Read the Segment Anything research paper for a detailed comparison of both techniques. {{light_callout_end}} Dive into SAM's Network Architecture and Design SAM’s design hinges on three main components: The promptable segmentation task to enable zero-shot generalization. The model architecture. The dataset that powers the task and model. Source Task SAM was trained on millions of images and over a billion masks to return a valid segmentation mask for any prompt. The prompt, in this case, is the segmentation task and can be foreground/background points, a rough box or mask, clicks, text, or, in general, any information indicating what to segment in an image. The task is also used as the pre-training objective for the model. Model SAM’s architecture comprises three components that work together to return a valid segmentation mask: An image encoder to generate one-time image embeddings. A prompt encoder that embeds the prompts. A lightweights mask decoder that combines the embeddings from the prompt and image encoders. Source We will dive into the architecture in the next section, but for now, let’s take a look at the dataset. Data Engine and Dataset A data engine is needed to power the tasks and improve the dataset and model. The data engine has three stages: Assisted-manual, where SAM assists annotators in annotating masks, similar to a classic interactive segmentation setup. Semi-automatic, where SAM can automatically generate masks for a subset of objects by prompting it with likely object locations, and annotators focus on annotating the remaining objects, helping increase mask diversity. Fully automatic, where human annotators prompt SAM with a regular grid of foreground points, yielding on average 100 high-quality masks per image. The data engine builds the large segment anything 1-billion mask dataset Meta AI released. A Quick Guide to the Segment Anything Model’s Structure Source Image Encoder At the highest level, an image encoder (a masked auto-encoder, MAE, pre-trained Vision Transformer, ViT) generates one-time image embeddings and can be applied prior to prompting the model. Prompt Encoder The prompt encoder encodes background points, masks, bounding boxes, or texts into an embedding vector in real time. The research considers two sets of prompts: sparse (points, boxes, text) and dense (masks).  Points and boxes are represented by positional encodings and added with learned embeddings for each prompt type. Free-form text prompts are represented with an off-the-shelf text encoder from CLIP. Dense prompts, like masks, are embedded with convolutions and summed element-wise with the image embedding. Mask Decoder A lightweight mask decoder predicts the segmentation masks based on the embeddings from both the image and prompt encoders. It maps the image embedding, prompt embeddings, and an output token to a mask. All of the embeddings are updated by the decoder block, which uses prompt self-attention and cross-attention in two directions (from prompt to image embedding and back). The masks are annotated and used to update the model weights. This layout enhances the dataset and allows the model to learn and improve over time, making it efficient and flexible. Segment Anything 1-Billion Mask Dataset The Segment Anything 1 Billion Mask (SA-1B) dataset is the largest labeled segmentation dataset to date. It is specifically designed for the development and evaluation of advanced segmentation models. We think the dataset will be an important part of training and fine-tuning future general-purpose models. This would allow them to achieve remarkable performance across diverse segmentation tasks. For now, the dataset is only available under a permissive license for research. The SA-1B dataset is unique due to its: Diversity Size High-quality annotations Diversity The dataset is carefully curated to cover a wide range of domains, objects, and scenarios, ensuring that the model can generalize well to different tasks. It includes images from various sources, such as natural scenes, urban environments, medical imagery, satellite images, and more.  This diversity helps the model learn to segment objects and scenes with varying complexity, scale, and context. Source Size The SA-1B dataset, which contains over a billion high-quality annotated images, provides ample training data for the model. The sheer volume of data helps the model learn complex patterns and representations, enabling it to achieve state-of-the-art performance on different segmentation tasks. Source High-quality Annotations The dataset has been carefully annotated with high-quality masks, leading to more accurate and detailed segmentation results. In the Responsible AI (RAI) analysis of the SA-1B dataset, potential fairness concerns and biases in geographic and income distribution were investigated. The research paper showed that SA-1B has a substantially higher percentage of images from Europe, Asia, and Oceania, as well as middle-income countries, compared to other open-source datasets. It's important to note that the SA-1B dataset features at least 28 million masks for all regions, including Africa. This is 10 times more than the total number of masks in any previous dataset. Source At Encord, we think the SA-1B dataset will enter the computer vision hall of fame (together with famous datasets such as COCO, ImageNet, and MNIST) as a resource for the development of future computer vision segmentation models. Is Segment Anything Model Open Source? The short answer is YES! The SA-1B Dataset has been released as open source for research purposes. In addition, Meta AI released the pre-trained models (~2.4 GB in size) and code under Apache 2.0 (a permissive license) following FAIR’s commitment to open research. It is freely accessible on GitHub. The training dataset is also available, alongside an interactive demo web UI. Source How to Fine-Tune Segment Anything Model (SAM) Now that you know the datasets SAM was trained on, you can identify if a dataset that represents your tasks was covered in the SA-1B dataset. If it’s not, or is underrepresented, you can consider finetuning SAM’s weights on your dataset. Yes, that’s possible with the pre-trained models that are also open sourced. As of the time of this writing, the Meta AI team has not added a way to fine-tune SAM specific applications, but Alex Bonnet, our Machine Learning Solutions Engineer, has prepared a step-by-step guide on how to do this. The steps include: Create a custom dataset by extracting the bounding box coordinates (prompts for the model), and extracting the ground truth segmentation masks. Prepare for fine-tuning by converting the input images to PyTorch tensors, a format SAM's internal functions expect. Run fine-tuning step by instantiating a training loop that focuses on using the lightweight mask decoder to iterate over the data items, generating masks, and comparing them to your ground truth masks. The mask decoder is easier, faster, and more memory efficient to fine tune. Compare your tuned model to the original model to make sure there’s indeed a significant improvement. {{light_callout_start}} To see the fine-tuning process in practice, check out our detailed blog, How To Fine-Tune Segment Anything, that also includes a Colab notebook as a walkthrough. {{light_callout_end}} How to Use the Segment Anything Model for AI-Assisted Labeling At Encord, we see the Segment Anything Model (SAM) as a game changer in AI-assisted labeling. It basically eliminates the need to go through the pain of segmenting images with polygon drawing tools and allows you to focus on the data tasks that are more important for your model.  These other data tasks include mapping the relationships between different objects, giving them attributes that describe how they act, and evaluating the training data to make sure it is balanced, diverse, and free of bias. Enhancing Manual Labeling with AI SAM can be used to create AI-assisted workflow enhancements and boost productivity for annotators. Here are just a few improvements we think SAM can contribute: Improved accuracy: Annotators can achieve more precise and accurate labels, reducing errors and improving the overall quality of the annotated data. Faster annotation: No doubt that SAM will speed up the labeling process, enabling annotators to complete tasks more quickly and efficiently when combined with a suitable image annotation tool. Consistency: Having all annotators use a version of SAM would ensure consistency across annotations, which is particularly important when multiple annotators are working on the same project. Reduced workload: By automating the segmentation of complex and intricate structures, SAM significantly reduces the manual workload for annotators, allowing them to focus on more challenging and intricate tasks. Continuous learning: As annotators refine and correct SAM's assisted labeling, we could implement it such that the model continually learns and improves, leading to better performance over time and further streamlining the annotation process. As such, integrating SAM into the annotation workflow is a no-brainer from our side, and it would allow our current and future customers to accelerate the development of cutting-edge computer vision applications. How can SAM Contributes to AI-Assisted Labeling To give an example of how SAM can contribute to AI-assisted labeling, consider the medical image example from before. We uploaded the DICOM image to the demo web UI, and spent 10 seconds clicking the image to segment the different areas of interest. Afterward, we did the same exercise with manual labeling using polygon annotations, which took 2.5 minutes. A 15x improvement in the labeling speed! We’re excited to start building this capability into Encord’s platform. Real-World Use Cases and Applications SAM can be used in almost every single segmentation task, from instance segmentation to panoptic segmentation. We’re excited about how quickly SAM can help you pre-label objects with almost pixel-perfect segmentation masks before your expert reviewer adds the ontology on top.  From agriculture and retail to medical imagery and geospatial imagery, the AI-assisted labeling that SAM can achieve is endless. It will be hard to imagine a world where SAM is not a default feature in all major annotation tools. This is why we at Encord are very excited about this new technology. Find other applications that could leverage SAM below.  Image and Video Editors SAM’s outstanding ability to provide accurate segmentation masks for even the most complex videos and images can provide image and video editing applications with automatic object segmentation skills. Whether the prompts (point coordinates, bounding boxes, etc.) are in the foreground or background, SAM uses positional encodings to indicate if the prompt is in the foreground or background. Generating Synthetic Datasets for Low-resource Industries One challenge that has plagued computer vision applications in industries like manufacturing is the lack of datasets. For example, industries building car parts and planning to detect defects in the parts along the production line cannot afford to gather large datasets for that use case. You can use SAM to generate synthetic datasets for your applications. If you realize SAM does not work particularly well for your applications, an option is to fine-tune it on existing datasets. {{light_callout_start}} Interested in synthetic data? Read our article What Is Synthetic Data Generation and Why Is It Useful? {{light_callout_end}} Gaze-based Segmentation AR applications can use SAM’s Zero-Shot Single Point Valid Mask Evaluation technique to segment objects through devices like AR glasses based on where subjects gaze. This can help AR technologies give users a more realistic sense of the world as they interact with those objects. Where Does This Leave Us? The Segment Anything Model (SAM) truly represents a groundbreaking development in the field of computer vision. By leveraging promptable segmentation tasks, SAM can adapt to a wide variety of downstream segmentation problems using prompt engineering.  This innovative approach, combined with the largest labeled segmentation dataset to date (SA-1B), allows SAM to achieve state-of-the-art performance in various segmentation tasks. With the potential to significantly enhance AI-assisted labeling and reduce manual labor in image segmentation tasks, SAM can pave the way in industries such as agriculture, retail, medical imagery, and geospatial imagery.  At Encord, we recognize the immense potential of SAM, and we are soon bringing the model to the Encord Platform to support AI-assisted labeling, further streamlining the data annotation process for users. As an open-source model, SAM will inspire further research and development in computer vision, encouraging the AI community to push the boundaries of what is possible in this rapidly evolving field.  Ultimately, SAM marks a new chapter in the story of computer vision, demonstrating the power of foundation models in transforming the way we perceive and understand the world around us. {{SAM_CTA}} Frequently Asked Questions on Segment Anything Model (SAM) How do I fine-tune SAM for my tasks? We have provided a step-by-step walkthrough you can follow to fine-tune SAM for your tasks. Check out the tutorial: How To Fine-Tune Segment Anything. What datasets were used to train SAM? The Segment Anything 1 Billion Mask (SA-1B) dataset has been touted as the “ImageNet of segmentation tasks.” The images vary across subject matter. Scenes, objects, and places frequently appear throughout the dataset. Masks range from large-scale objects such as buildings to fine-grained details like door handles. See the data card and dataset viewer to learn more about the composition of the dataset. Does SAM work well for all tasks? Yes. You can automatically select individual items from images—it works very well on complex images. SAM is a foundation model that provides multi-task capabilities to any application you plug it into. Does SAM work well for ambiguous images? Yes, it does. Because of this, you might find duplicates in your mask sets when you run SAM over your dataset. This allows you to select the most appropriate masks for your task. In this case, you should add a post-processing step to segment the most suitable masks for your task. How long does it take SAM to generate segmentation masks? SAM can generate a segment in as little as 50 milliseconds—practically real-time! Do I need a GPU to run SAM? Although it is possible to run SAM on your CPU, you can use a GPU to achieve significantly faster results from SAM.

Computer vision is having its ChatGPT moment with the release of the Segment Anything Model (SAM) by Meta last week. Trained over 11 billion segmentation masks, SAM is a foundation model for predictive AI use cases rather than generative AI. While it has shown an incredible amount of flexibility in its ability to segment over wide-ranging image modalities and problem spaces, it was released without “fine-tuning” functionality. This tutorial will outline some of the key steps to fine-tune SAM using the mask decoder, particularly describing which functions from SAM to use to pre/post process the data so that it's in a good shape for fine tuning. {{Training_data_CTA::Supercharge your annotations by fine-tuning SAM for your use case}} What is the Segment Anything Model (SAM)? The Segment Anything Model (SAM) is a segmentation model developed by Meta AI. It is considered the first foundational model for Computer Vision. SAM was trained on a huge corpus of data containing millions of images and billions of masks, making it extremely powerful. As its name suggests, SAM is able to produce accurate segmentation masks for a wide variety of images. SAM’s design allows it to take human prompts into account, making it particularly powerful for Human In The Loop annotation. These prompts can be multi-modal: they can be points on the area to be segmented, a bounding box around the object to be segmented or a text prompt about what should be segmented. The model is structured into 3 components: an image encoder, a prompt encoder and a mask decoder. Source The image encoder generates an embedding for the image being segmented, whilst the prompt encoder generates an embedding for the prompts. The image encoder is a particularly large component in the model. This is in contrast to the lightweight mask   decoder, which predicts segmentation masks based on the embeddings. Meta AI has made the weights and biases of the model trained on the Segment Anything 1 Billion Mask (SA-1B) dataset available as a model checkpoint. {{light_callout_start}} Learn more about how Segment Anything works in our explainer blog post Segment Anything Model (SAM) Explained. {{light_callout_end}} What is Model Fine-Tuning? Publicly available state of the art models have a custom architecture and are typically supplied with pre-trained model weights. If these architectures were supplied without weights then the models would need to be trained from scratch by the users, who would need to use massive datasets to obtain state of the art performance. Model fine tuning is the process of taking a pre-trained model (architecture+weights) and showing it data for a particular use case. This will typically be data that the model hasn’t seen before, or that is underrepresented in its original training dataset. The difference between fine tuning the model and starting from scratch is the starting value of the weights and biases. If we were training from scratch, these would be randomly initialised according to some strategy. In such a starting configuration, the model would ‘know nothing’ of the task at hand and perform poorly. By using pre existing weights and biases as a starting point we can ‘fine tune’ the weights and biases so that our model works better on our custom dataset. For example: the information learnt to recognise cats (edge detection, counting paws) will be useful for recognising dogs. Why Would I Fine-Tune a Model? The purpose of fine tuning a model is to obtain higher performance on data which the pre-trained model has not seen before. For example, an image segmentation model trained on a broad corpus of data gathered from phone cameras will have mostly seen images from a horizontal perspective. If we tried to use this model for satellite imagery taken from a vertical perspective, it may not perform as well. If we were trying to segment rooftops, the model may not yield the best results. The pre-training is useful because the model will have learnt how to segment objects in general, so we want to take advantage of this starting point to build a model which can accurately segment rooftops. Furthermore, it is likely that our custom dataset would not have millions of examples, so we want to fine tune instead of training the model from scratch. Fine tuning is desirable so that we can obtain better performance on our specific use case, without having to incur the computational cost of training a model from scratch. How to Fine-Tune Segment Anything Model [With Code] Background & Architecture We gave an overview of the SAM architecture in the introduction section. The image encoder has a complex architecture with many parameters. In order to fine tune the model, it makes sense for us to focus on the mask decoder which is lightweight and therefore easier, faster and more memory efficient to fine tune. In order to fine tune SAM, we need to extract the underlying pieces of its architecture (image and prompt encoders, mask decoder). We cannot use SamPredictor.predict (link) for two reasons: We want to fine tune only the mask decoder This function calls SamPredictor.predict_torch which has the  @torch.no_grad() decorator (link), which prevents us from computing gradients Thus, we need to examine the SamPredictor.predict function and call the appropriate functions with gradient calculation enabled on the part we want to fine tune (the mask decoder). Doing this is also a good way to learn more about how SAM works. Creating a Custom Dataset We need three things to fine tune our model: Images on which to draw segmentations Segmentation ground truth masks Prompts to feed into the model We chose the stamp verification dataset (link) since it has data which SAM may not have seen in its training (i.e., stamps on documents). We can verify that it performs well, but not perfectly, on this dataset by running inference with the pre-trained weights. The ground truth masks are also extremely precise, which will allow us to calculate accurate losses. Finally, this dataset contains bounding boxes around the segmentation masks, which we can use as prompts to SAM. An example image is shown below. These bounding boxes align well with the workflow that a human annotator would go through when looking to generate segmentations. Input Data Preprocessing We need to preprocess the scans from numpy arrays to pytorch tensors. To do this, we can follow what happens inside SamPredictor.set_image (link) and SamPredictor.set_torch_image (link) which preprocesses the image. First, we can use utils.transform.ResizeLongestSide to resize the image, as this is the transformer used inside the predictor (link). We can then convert the image to a pytorch tensor and use the SAM preprocess method (link) to finish preprocessing. Training Setup We download the model checkpoint for the vit_b model and load them in: sam_model = sam_model_registry['vit_b'](checkpoint='sam_vit_b_01ec64.pth') We can set up an Adam optimizer with defaults and specify that the parameters to tune are those of the mask decoder: optimizer = torch.optim.Adam(sam_model.mask_decoder.parameters())  At the same time, we can set up our loss function, for example Mean Squared Error loss_fn = torch.nn.MSELoss() Training Loop In the main training loop, we will be iterating through our data items, generating masks and comparing them to our ground truth masks so that we can optimise the model parameters based on the loss function. In this example we used a GPU for training since it is much faster than using a CPU. It is important to use .to(device) on the appropriate tensors to make sure that we don’t have certain tensors on the CPU and others on the GPU. We want to embed images by wrapping the encoder in the torch.no_grad() context manager, since otherwise we will have memory issues, along with the fact that we are not looking to fine tune the image encoder. with torch.no_grad(): image_embedding = sam_model.image_encoder(input_image) We can also generate the prompt embeddings within the no_grad context manager. We use our bounding box coordinates, converted to pytorch tensors. with torch.no_grad(): sparse_embeddings, dense_embeddings = sam_model.prompt_encoder( points=None, boxes=box_torch, masks=None, ) Finally, we can generate the masks. Note that here we are in single mask generation mode (in contrast to the 3 masks that are normally output). low_res_masks, iou_predictions = sam_model.mask_decoder( image_embeddings=image_embedding, image_pe=sam_model.prompt_encoder.get_dense_pe(), sparse_prompt_embeddings=sparse_embeddings, dense_prompt_embeddings=dense_embeddings, multimask_output=False, ) The final step here is to upscale the masks back to the original image size, since they are low resolution. We can use Sam.postprocess_masks to achieve this. We will also want to generate binary masks from the predicted masks so that we can compare these to our ground truths. It is important to use torch functionals in order to not break backpropagation. upscaled_masks = sam_model.postprocess_masks(low_res_masks, input_size, original_image_size).to(device) from torch.nn.functional import threshold, normalize binary_mask = normalize(threshold(upscaled_masks, 0.0, 0)).to(device) Finally we can calculate the loss and run an optimisation step: loss = loss_fn(binary_mask, gt_binary_mask) optimizer.zero_grad() loss.backward() optimizer.step() By repeating this over a number of epochs and batches we can fine tune the SAM decoder. Saving Checkpoints and Starting a Model from it Once we are done with training and satisfied by the performance uplift, we can save the state dict of the tuned model using: torch.save(model.state_dict(), PATH) We can then load this state dict when we want to perform inference on data that is similar to the data we used to fine tune the model. {{light_callout_start}} You can find the Colab Notebook with all the code you need to fine-tune SAM here. Keep reading if you want a fully working solution out of the box! {{light_callout_end}} Fine-Tuning for Downstream Applications While SAM does not currently offer fine-tuning out of the box, we are building a custom fine tuner integrated with the Encord platform. As shown in this post, we fine tune the decoder in order to achieve this. This is available as an out of the box one click procedure in the web app, where the hyperparameters are automatically set. Original vanilla SAM mask: Mask generated by fine tuned version of the model: We can see that this mask is tighter than the original mask. This was the result of fine tuning on a small subset of images from the stamp verification dataset, and then running the tuned model on a previously unseen example. With further training and more examples we could obtain even better results. Conclusion That's all, folks! You have now learned how to fine-tune the Segment Anything Model (SAM). If you're looking to fine-tune SAM out of the box, you might also be interested to learn that we have recently released the Segment Anything Model in Encord, allowing you to fine-tune the model without writing any code. {{SAM_CTA}}

Discover the 9 most popular free and paid image annotation tools to know about heading into 2024. Compare their features and pricing, and choose the best image labelling tool for your use case. Annotating images is still one of the most time-consuming steps in bringing a computer vision project to market. To help you out, we compiled a list of the most popular image annotation tools out there. Whether you are: A computer vision team building unmanned drones with your own in-house annotation tool A data science team working on a large-scale defect inspection model looking for labelling services Or a data operations team looking for the right platform for your radiologists to accurately label CT scans This guide will help you compare the top image annotation tools and find the right one for you. We will compare each based on key factors - including image annotation functionality, support for different data types and use cases, QA/QC capabilities, security and data privacy, data management, integration with the machine learning pipeline, and customer support. We’ll be updating this review periodically with notable feature releases and the latest image annotation tools coming to market. So let's get to it! The 9 most popular image annotation tools:  Encord Annotate Scale CVAT Labelbox Playment Appen Dataloop V7 Labs Hive {{light_callout_start}} Exploring image annotation tools? Label data 10x faster and gain control of your training data with Encord - get in touch to start today. {{light_callout_end}} Encord Annotate Encord Annotate is an automated annotation platform for AI-assisted image annotation, video annotation, and dataset management. It's the best option for teams that are: Looking for automated, semi-automated or AI-assisted image and video annotation. Annotating all modalities (DICOM and NIfTI, SAR, ultra-high-resolution, and more). Wanting one place to easily manage annotators, track performance, and create QA/QC workflows. Benefits & Key features: Use-case-centric annotations — from native DICOM & NIfTI annotations for medical imaging to SAR-specific features for geospatial data. Support for all annotation types — bounding boxes, polygons, polylines, image segmentation, keypoint, custom object primitives (rotating bounding boxes, 3D cuboids), and more. Incorporates auto-annotation tools such as Meta's Segment Anything Model and other AI-assisted labeling techniques. Integrated data labeling services. Integrated MLOps workflows for computer vision and machine learning teams — to detect edge cases and gaps in your training data and generate augmented data to improve label quality. Easy collaboration, annotator management, and QA workflows — to track annotator performance and increase label quality. Robust security functionality — label audit trails, encryption, FDA, CE Compliance, and HIPAA compliance. Advanced Python SDK and API access (+ easy export into JSON and COCO formats). Best for teams who: Are graduating from an in-house solution or open-source tool and need a robust, secure, and collaborative platform to scale your annotation workflows. Haven't found an annotation platform that can actually support their use case as well as they'd like (such as building complex nested ontologies or rendering DICOM formats natively). Pricing: Free trial model with simple per-user pricing thereafter {{Training_data_CTA::Label data 10x faster and gain control of your training data}} Scale Scale AI, now Scale, is a data and labeling services platform that supports image, audio, text, and video annotations. As of 2023, Scale has also expanded into new use cases, like user experience optimization, large language models, and synthetic data. Benefits & Key features: Leader in workforce management Support for multiple data modalities (image, video, document processing, audio and more) Nucleus dataset management Best for: Workforce management Pricing: On a per-image basis 💡 More insights on data labeling with Scale: Teams looking for annotation tools for Autonomous Vehicle vision should know… Scale is one of the earliest platforms on the market to support 3D Sensor Fusion annotation for RADAR and LiDAR use cases, and their capabilities are some of the most widely adopted. Teams looking for medical imaging annotation tools should know… Platforms like Scale will usually not support DICOM or NIfTI data types nor allow companies to work with their own annotators on the platform. So if you’re looking to annotate medical images, like CT, X-Ray, or MRI scans, you’ll want to look for a medical purpose-built platform (built for radiologist or physician annotators and specialist formats like DICOM and NIfTI) like Annotate. Teams looking for labeling services should know… Scale is a very popular option for data labeling services. Other alternatives (discussed later in this article) are Appen and Playment. CVAT CVAT (Computer Vision Annotation Tool) is a free, open-source, web-based annotation toolkit built by Intel. For image labeling - CVAT supports four types of annotations (points, polygons, bounding boxes, and polylines), as well as a subset of computer vision tasks (image segmentation, object detection, and image classification). In 2022, CVAT’s data, content, and GitHub repository were migrated over to OpenCV, where CVAT continues to be open-source. Benefits & Key features: Easy (and free) to get started labeling images with Great for manual data annotation — also supports semi-assisted labeling Robust ground-level annotation capabilities (including classification and object detection) for a broad set of computer vision use cases Best for: Students, researchers, and academics testing the waters with image annotation (perhaps with a few images or a small dataset) Pricing: Free! 💡 More insights on image labeling with CVAT: If your team is looking for a free annotation tool, you should know… CVAT is one of the most popular open-source tools in the space, with over 1 million downloads since 2021 — other popular free image annotation alternatives to CVAT are 3D Slicer, Labelimg, VoTT (Visual Object Tagging Tool - developed by Microsoft), VIA (VGG Image Annotator), LabelMe, and Label Studio. If data security is a requirement for your annotation project… Commercial labeling tools will most likely be a better fit — key security features like audit trails, encryption, SSO, and generally-required vendor certifications (like SOC2, HIPAA, FDA, and GDPR) are usually not available in open-source tools. Further reading: Buy vs build for computer vision annotation - what's better? Overview of open source annotation tools for computer vision Complete guide to image annotation for computer vision Labelbox Labelbox is a US-based data annotation platform founded in 2017. Like most of the other platforms mentioned in this guide, Labelbox offers both an image labelling platform, as well as labelling services. Teams can annotate a wide range of data types (PDF, audio, images, videos, and more). Benefits & Key features: Flexible QA workflows and annotator performance tracking Integrated 3rd party labeling services through Labelbox Boost Model-assisted annotations Robust support for multiple data types beyond images, especially text Best for: Teams looking for a powerful platform to quickly annotate documents and text Pricing: Varies based on the volume of data, percent of the total volume needing to be labeled, number of seats, number of projects, and percent of data used in model training. 💡 More insights on image labeling with Labelbox: Teams looking for document processing annotation should know that… Labelbox has been heavily investing in its Document and Conversational AI products, released in 2023. Its document annotation capabilities are quickly gaining traction, particularly within the financial services industry.  Teams carrying out annotation projects that are use-case specific should know that… As generalist tools, platforms like Labelbox are great at handling a broad variety of data types. If you’re working on a unique use-case-specific annotation project (like scans in DICOM formats or high-resolution images that require pixel-perfect annotations), other commercial image annotation platforms will be a better fit. Playment Playment is a fully-managed data annotation platform. The workforce labeling company was acquired by Telus in 2021 and provides computer vision teams with high-quality training data for various use cases, supported by manual labelers and a machine learning platform. Benefits & Key features: One of the largest global workforces of contractors and data labelers (+1M annotators) Human-assisted 2D and 3D functionality for image annotations Speech recognition training platform (handles all data types across 500+ languages and dialects) Best for: Teams looking for a fully managed solution to get labeled data from (just share data and label guidelines) Pricing: Enterprise plan Appen Appen is a data labeling services platform founded in 1996, making it one of the first and oldest solutions in the market. The company offers data labeling services for a wide range of industries and in 2019, acquired Figure Eight to build out its software capabilities and help businesses also train and improve their computer vision models. Benefits & Key features: Support for multiple annotation types (bounding boxes, polygons, and image segmentation) Data sourcing (pre-labeled datasets), data preparation, and real-world model evaluation Natural language processing and functionality for broader text-to-speech support are available Best for: Teams looking for image data sourcing and collection alongside annotation services Pricing: Enterprise plan Dataloop Dataloop is an Israel-based data labeling platform that provides a comprehensive solution for data management and annotation projects. The tool offers data labeling capabilities across images, text, audio, and video annotation, helping businesses train and improve their machine learning models. Benefits & Key features: Features for image annotation tasks, including classification, detection, and semantic segmentation Support for video annotations Intuitive and easy-to-use user interface Best for: Teams looking for a powerful platform to annotate wide variety of data types Pricing: Free trial and an enterprise plan 💡 More insights on image labeling with Dataloop: Teams carrying out specific image and video annotation projects that are use-case specific should know that… As generalist tools, platforms like Dataloop are built to support a wide variety of simple use cases, so other commercial platforms are a better fit if you’re trying to label use-case-specific annotation projects (like high-resolution images that require pixel-perfect annotations in satellite imaging or DICOM files for medical teams).  V7 Labs V7 is a UK-based data annotation platform founded in 2018. The company enables teams to annotate training data, support the human-in-the-loop processes, and also connect with annotation services. V7 offers annotation of a wide range of data types alongside image annotation tooling, including documents and videos. Benefits & Key features: Robust project management and automation workflow functionality, with real-time collaboration and tagging Integrated labeling services Model-assisted annotation of multiple annotation types (segmentation, detection, and more) Best for: Students or teams looking for a generalist platform to easily annotate different data types in one place (like documents, images, and short videos) Pricing: Various options, including academic, business, and pro Hive Hive was founded in 2013 and provides cloud-based AI solutions for companies wanting to label content across a wide range of data types, including images, video, audio, text, and more. Hive’s platform allows engineers to bring their own pre-trained ML models into the platform, and leverage them for content moderation (including detection of AI-generated data). Benefits & Key features: End-to-end image annotation tool Support for unique image annotation use cases (ad targeting, semi-automated logo detection) Flexible access to model predictions with a single API call Bring your own AI model (BYOM) Best for: Teams labeling images and other data types for the purpose of content moderation. Hive is especially popular with social media platforms like Reddit, BeReal, and Quora. Pricing: Enterprise plan Conclusion There you have it! The 9 Best Image Annotation Tools for computer vision in 2023.  For further reading, you might also want to check out a few 2023 honorable mentions, both paid and free annotation tools: Labelstudio - user-friendly open source tool praised for its manual annotation process capabilities. Supervisely - commercial data labeling platform praised for its quality control functionality and basic interpolation feature. VoTT - open source tool, praised for its tags and assets export to Tensorflow (PascalVOC) and YOLO format. {{try_encord_CTA_annotate_visual}}