An Introduction to Diffusion Models for Machine Learning

Update: Segment Anything (SAM) is live in Encord! Check out our tutorial on How To Fine-Tune Segment Anything or book a demo with the team to learn how to use the Segment Anything Model (SAM) to reduce labeling costs. If you thought the AI space was already moving fast with ChatGPT, GPT4, and Stable Diffusion, then strap in and prepare for the next groundbreaking innovation in AI. Meta’s FAIR lab has just released the Segment Anything Model (SAM), a state-of-the-art image segmentation model that aims to change the field of computer vision.  SAM is based on foundation models that have significantly impacted natural language processing (NLP). It focuses on promptable segmentation tasks, using prompt engineering to adapt to diverse downstream segmentation problems. In this blog post, you will:  Learn how it compares to other foundation models. Learn about the Segment Anything (SA) project Dive into SAM's network architecture, design, and implementation. Learn how to implement and fine-tune SAM Discover potential uses of SAM for AI-assisted labeling. Why Are We So Excited About SAM?  Having tested it out for a day now, we can see the following incredible advances: SAM can segment objects by simply clicking or interactively selecting points to include or exclude from the object. You can also create segmentations by drawing bounding boxes or segmenting regions with a polygon tool, and it will snap to the object. When encountering uncertainty in identifying the object to be segmented, SAM can produce multiple valid masks. SAM can identify and generate masks for all objects present in an image automatically. After precomputing the image embeddings, SAM can provide a segmentation mask for any prompt instantly for real-time interaction with the model. A Brief History of Meta's AI & Computer Vision As one of the leading companies in the field of artificial intelligence (AI), Meta has been pushing the boundaries of what's possible with machine learning models: from recently released open source models such as LLaMA to developing the most used Python library for ML and AI, PyTorch. Advances in Computer Vision Computer vision has also experienced considerable advancements, with models like CLIP bridging the gap between text and image understanding. These models use contrastive learning to map text and image data. This allows them to generalize to new visual concepts and data distributions through prompt engineering.  FAIR’s Segment Anything Model (SAM) is the latest breakthrough in this field. Their goal was to create a foundation model for image segmentation that can adapt to various downstream tasks using prompt engineering. The release of SAM started a wave of vision foundation models and vision language models like LLaVA, GPT-4 vision, Gemini, and many more. Let’s briefly explore some key developments in computer vision that have contributed to the growth of AI systems like Meta's. Convolutional Neural Networks (CNNs) CNNs, first introduced by Yann LeCun (now VP & Chief AI Scientist at Meta) in 1989, have emerged as the backbone of modern computer vision systems, enabling machines to learn and recognize complex patterns in images automatically.  By employing convolutional layers, CNNs can capture local and global features in images, allowing them to recognize objects, scenes, and actions effectively. This has significantly improved tasks such as image classification, object detection, and semantic segmentation. Interested in learning more about CNNs? Read our Convolutional Neural Networks (CNN) Overview. Generative Adversarial Networks (GANs) GANs are a type of deep learning model that Ian Goodfellow and his team came up with in 2014. They are made up of two neural networks, a generator, and a discriminator, competing. The generator aims to create realistic outputs, while the discriminator tries to distinguish between real and generated outputs. The competition between these networks has resulted in the creation of increasingly realistic synthetic images. It has led to advances in tasks such as image synthesis, data augmentation, and style transfer. Transfer Learning and Pre-trained Models Like NLP, computer vision has benefited from the development of pre-trained models that can be fine-tuned for specific tasks. Models such as ResNet, VGG, and EfficientNet have been trained on large-scale image datasets, allowing researchers to use these models as a starting point for their own projects. The Growth of Foundation Models Foundation models in natural language processing (NLP) have made significant strides in recent years, with models like Meta’s own LLaMA or OpenAI’s GPT-4 demonstrating remarkable capabilities in zero-shot and few-shot learning.  These models are pre-trained on vast amounts of data and can generalize to new tasks and data distributions by using prompt engineering. Meta AI has been instrumental in advancing this field, fostering research and the development of large-scale NLP models that have a wide range of applications. Here, we explore the factors contributing to the growth of foundation models. Large-scale Language Models The advent of large-scale language models like GPT-4 has been a driving force behind the development of foundation models in NLP. These models employ deep learning architectures with billions of parameters to capture complex patterns and structures in the training data.  Transfer Learning A key feature of foundation models in NLP is their capacity for transfer learning. Once trained on a large corpus of data, they can be fine-tuned on smaller, task-specific datasets to achieve state-of-the-art performance across a variety of tasks. Zero-shot and Few-shot Learning Foundation models have also shown promise in zero-shot and few-shot learning, where they can perform tasks without fine-tuning or with minimal task-specific training data. This capability is largely attributed to the models' ability to understand and generate human-like responses based on the context provided by prompts. Multi-modal Learning Another growing area of interest is multi-modal learning, where foundation models are trained to understand and generate content across different modalities, such as text and images.  Models like Gemini, GPT-4V, LLaVA, CLIP, and ALIGN show how NLP and computer vision could be used together to make multi-modal models that can translate actions from one domain to another. Read the blog Google Launches Gemini, Its New Multimodal AI Model to learn about the recent advancements in multimodal learning. Ethical Considerations and Safety The growth of foundation models in NLP has also raised concerns about their ethical implications and safety. Researchers are actively exploring ways to mitigate potential biases, address content generation concerns, and develop safe and controllable AI systems. Proof of this was the recent call for a six-month halt on all development of cutting-edge models. Segment Anything Model Vs. Previous Foundation Models SAM is a big step forward for AI because it builds on the foundations that were set by earlier models. SAM can take input prompts from other systems, such as, in the future, taking a user's gaze from an AR/VR headset to select an object, using the output masks for video editing, abstracting 2D objects into 3D models, and even popular Google Photos tasks like creating collages. It can handle tricky situations by generating multiple valid masks where the prompt is unclear. Take, for instance, a user’s prompt for finding Waldo: Image displaying semantic segmentations by the Segment Anything Model (SAM) One of the reasons the results from SAM are groundbreaking is because of how good the segmentation masks are compared to other techniques like ViTDet. The illustration below shows a comparison of both techniques: Segmentation masks by humans, ViTDet, and the Segment Anything Model (SAM) Read the Segment Anything research paper for a detailed comparison of both techniques. What is the Segment Anything Model (SAM)? SAM, as a vision foundation model, specializes in image segmentation, allowing it to accurately locate either specific objects or all objects within an image. SAM was purposefully designed to excel in promptable segmentation tasks, enabling it to produce accurate segmentation masks based on various prompts, including spatial or textual clues that identify specific objects. Is Segment Anything Model Open Source? The short answer is YES! The SA-1B Dataset has been released as open source for research purposes. In addition, Meta AI released the pre-trained models (~2.4 GB in size) and code under Apache 2.0 (a permissive license) following FAIR’s commitment to open research. It is freely accessible on GitHub. The training dataset is also available, alongside an interactive demo web UI. FAIR Segment Anything (SA) Paper How does the Segment Anything Model (SAM) work? SAM's architectural design allows it to adjust to new image distributions and tasks seamlessly, even without prior knowledge, a capability referred to as zero-shot transfer. Utilizing the extensive SA-1B dataset, comprising over 11 million meticulously curated images with more than 1 billion masks, SAM has demonstrated remarkable zero-shot performance, often surpassing previous fully supervised results. Before we dive into SAM’s network design, let’s discuss the SA project, which led to the introduction of the very first vision foundation model. Segment Anything Project Components The Segment Anything (SA) project is a comprehensive framework for image segmentation featuring a task, model, and dataset.  The SA project starts with defining a promptable segmentation task with broad applicability, serving as a robust pretraining objective, and facilitating various downstream applications. This task necessitates a model capable of flexible prompting and the real-time generation of segmentation masks to enable interactive usage. Training the model effectively required a diverse, large-scale dataset. The SA project implements a "data engine" approach to address the lack of web-scale data for segmentation. This involves iteratively using our efficient model to aid in data collection and utilizing the newly gathered data to enhance the model's performance. Let’s look at the individual components of the SA project. Segment Anything Task  The Segment Anything Task draws inspiration from natural language processing (NLP) techniques, particularly the next token prediction task, to develop a foundational model for segmentation. This task introduces the concept of promptable segmentation, wherein a prompt can contain various ,input forms such as foreground/background points, bounding boxes, masks, or free-form text, indicating what objects to segment in an image. The objective of this task is to generate valid segmentation masks based on any given prompt, even in ambiguous scenarios. This approach enables a natural pre-training algorithm and facilitates zero-shot transfer to downstream segmentation tasks by engineering appropriate prompts.  Using prompts and composition techniques allows for extensible model usage across a wide range of applications, distinguishing it from interactive segmentation models designed primarily for human interaction. Dive into SAM's Network Architecture and Design SAM’s design hinges on three main components: The promptable segmentation task to enable zero-shot generalization. The model architecture. The dataset that powers the task and model. Leveraging concepts from Transformer vision models, SAM prioritizes real-time performance while maintaining scalability and powerful pretraining methods. Scroll down to learn more about SAM’s network design. Foundation model architecture for the Segment Anything (SA) model Task SAM was trained on millions of images and over a billion masks to return a valid segmentation mask for any prompt. In this case, the prompt is the segmentation task and can be foreground/background points, a rough box or mask, clicks, text, or, in general, any information indicating what to segment in an image. The task is also used as the pre-training objective for the model. Model SAM’s architecture comprises three components that work together to return a valid segmentation mask: An image encoder to generate one-time image embeddings. A prompt encoder that embeds the prompts. A lightweight mask decoder that combines the embeddings from the prompt and image encoders. Components of the Segment Anything (SA) model We will dive into the architecture in the next section, but for now, let’s take a look at the data engine. Segment Anything Data Engine The Segment Anything Data Engine was developed to address the scarcity of segmentation masks on the internet and facilitate the creation of the extensive SA-1B dataset containing over 1.1 billion masks. A data engine is needed to power the tasks and improve the dataset and model. The data engine has three stages: Model-assisted manual annotation, where professional annotators use a browser-based interactive segmentation tool powered by SAM to label masks with foreground/background points. As the model improves, the annotation process becomes more efficient, with the average time per mask decreasing significantly. Semi-automatic, where SAM can automatically generate masks for a subset of objects by prompting it with likely object locations, and annotators focus on annotating the remaining objects, helping increase mask diversity. The focus shifts to increasing mask diversity by detecting confident masks and asking annotators to annotate additional objects. This stage contributes significantly to the dataset, enhancing the model's segmentation capabilities. Fully automatic, where human annotators prompt SAM with a regular grid of foreground points, yielding on average 100 high-quality masks per image. The annotation becomes fully automated, leveraging model enhancements and techniques like ambiguity-aware predictions and non-maximal suppression to generate high-quality masks at scale. This approach enables the creation of the SA-1B dataset, consisting of 1.1 billion high-quality masks derived from 11 million images, paving the way for advanced research in computer vision. Segment Anything 1-Billion Mask Dataset The Segment Anything 1 Billion Mask (SA-1B) dataset is the largest labeled segmentation dataset to date. It is specifically designed for the development and evaluation of advanced segmentation models. We think the dataset will be an important part of training and fine-tuning future general-purpose models. This would allow them to achieve remarkable performance across diverse segmentation tasks. For now, the dataset is only available under a permissive license for research. The SA-1B dataset is unique due to its: Diversity Size High-quality annotations Diversity The dataset is carefully curated to cover a wide range of domains, objects, and scenarios, ensuring that the model can generalize well to different tasks. It includes images from various sources, such as natural scenes, urban environments, medical imagery, satellite images, and more.  This diversity helps the model learn to segment objects and scenes with varying complexity, scale, and context. Distribution of images and masks for training Segment Anything (SA) model Size The SA-1B dataset, which contains over a billion high-quality annotated images, provides ample training data for the model. The sheer volume of data helps the model learn complex patterns and representations, enabling it to achieve state-of-the-art performance on different segmentation tasks. Relative size of SA-1B to train the Segment Anything (SA) model High-Quality Annotations The dataset has been carefully annotated with high-quality masks, leading to more accurate and detailed segmentation results. In the Responsible AI (RAI) analysis of the SA-1B dataset, potential fairness concerns and biases in geographic and income distribution were investigated. The research paper showed that SA-1B has a substantially higher percentage of images from Europe, Asia, and Oceania, as well as middle-income countries, compared to other open-source datasets. It's important to note that the SA-1B dataset features at least 28 million masks for all regions, including Africa. This is 10 times more than any previous dataset's total number of masks. Distribution of the images to train the Segment Anything (SA) model At Encord, we think the SA-1B dataset will enter the computer vision hall of fame (together with famous datasets such as COCO, ImageNet, and MNIST) as a resource for developing future computer vision segmentation models. Notably, the dataset includes downscaled images to mitigate accessibility and storage challenges while maintaining significantly higher resolution than many existing vision datasets. The quality of the segmentation masks is rigorously evaluated, with automatic masks deemed high quality and effective for training models, leading to the decision to include automatically generated masks exclusively in SA-1B. Let's delve into the SAM's network architecture, as it's truly fascinating. Segment Anything Model's Network Architecture The Segment Anything Model (SAM) network architecture contains three crucial components: the Image Encoder, the Prompt Encoder, and the Mask Decoder. Architecture of the Segment Anything (SA) universal segmentation model Image Encoder At the highest level, an image encoder (a masked auto-encoder, MAE, pre-trained Vision Transformer, ViT) generates one-time image embeddings. It is applied before prompting the model. The image encoder, based on a masked autoencoder (MAE) pre-trained Vision Transformer (ViT), processes high-resolution inputs efficiently. This encoder runs once per image and can be applied before prompting the model for seamless integration into the segmentation process. Prompt Encoder The prompt encoder encodes background points, masks, bounding boxes, or texts into an embedding vector in real-time. The research considers two sets of prompts: sparse (points, boxes, text) and dense (masks).  Points and boxes are represented by positional encodings and added with learned embeddings for each prompt type. Free-form text prompts are represented with an off-the-shelf text encoder from CLIP. Dense prompts, like masks, are embedded with convolutions and summed element-wise with the image embedding. The prompt encoder manages both sparse and dense prompts. Dense prompts, like masks, are embedded using convolutions and then summed element-wise with the image embedding. Sparse prompts, representing points, boxes, and text, are encoded using positional embeddings for each prompt type. Free-form text is handled with a pre-existing text encoder from CLIP. This approach ensures that SAM can effectively interpret various types of prompts, enhancing its adaptability. Mask Decoder A lightweight mask decoder predicts the segmentation masks based on the embeddings from both the image and prompt encoders. The mask decoder efficiently maps the image and prompt embeddings to generate segmentation masks. It maps the image embedding, prompt embeddings, and an output token to a mask. SAM uses a modified decoder block and then a dynamic mask prediction head, taking inspiration from already existing Transformer decoder blocks. This design incorporates prompt self-attention and cross-attention mechanisms to update all embeddings effectively. All of the embeddings are updated by the decoder block, which uses prompt self-attention and cross-attention in two directions (from prompt to image embedding and back). SAM's Mask Decoder is optimized for real-time performance, enabling seamless interactive prompting of the model. The masks are annotated and used to update the model weights. This layout enhances the dataset and allows the model to learn and improve over time, making it efficient and flexible. SAM's overall design prioritizes efficiency, with the prompt encoder and mask decoder running seamlessly in web browsers in approximately 50 milliseconds, enabling real-time interactive prompting. How to Use the Segment Anything Model? At Encord, we see the Segment Anything Model (SAM) as a game changer in AI-assisted labeling. It basically eliminates the need to go through the pain of segmenting images with polygon drawing tools and allows you to focus on the data tasks that are more important for your model.  These other data tasks include mapping the relationships between different objects, giving them attributes that describe how they act, and evaluating the training data to ensure it is balanced, diverse, and bias-free. Enhancing Manual Labeling with AI SAM can be used to create AI-assisted workflow enhancements and boost productivity for annotators. Here are just a few improvements we think SAM can contribute: Improved accuracy: Annotators can achieve more precise and accurate labels, reducing errors and improving the overall quality of the annotated data. Faster annotation: There is no doubt that SAM will speed up the labeling process, enabling annotators to complete tasks more quickly and efficiently when combined with a suitable image annotation tool. Consistency: Having all annotators use a version of SAM would ensure consistency across annotations, which is particularly important when multiple annotators are working on the same project. Reduced workload: By automating the segmentation of complex and intricate structures, SAM significantly reduces the manual workload for annotators, allowing them to focus on more challenging and intricate tasks. Continuous learning: As annotators refine and correct SAM's assisted labeling, we could implement it such that the model continually learns and improves, leading to better performance over time and further streamlining the annotation process. As such, integrating SAM into the annotation workflow is a no-brainer from our side, and it would allow our current and future customers to accelerate the development of cutting-edge computer vision applications. AI-Assisted Labeling with Segment Anything Model on Encord Consider the medical image example from before to give an example of how SAM can contribute to AI-assisted labeling. We uploaded the DICOM image to the demo web UI, and spent 10 seconds clicking the image to segment the different areas of interest. Afterward, we did the same exercise with manual labeling using polygon annotations, which took 2.5 minutes. A 15x improvement in the labeling speed! Read the blog How to use SAM to Automate Data Labeling in Encord for more information.   Combining SAM with Encord Annotate harnesses SAM's versatility in segmenting diverse content alongside Encord's robust ontologies, interactive editing capabilities, and extensive media compatibility. Encord seamlessly integrates SAM for annotating images, videos, and specialized data types like satellite imagery and DICOM files. This includes support for various medical imaging formats, such as X-ray, CT, and MRI, requiring no extra input from the user. How to Fine-Tune Segment Anything Model (SAM) To fine-tune the SAM model effectively, particularly considering its intricate architecture, a pragmatic strategy involves refining the mask decoder. Unlike the image encoder, which boasts a complex architecture with numerous parameters, the mask decoder offers a streamlined and lightweight design. This characteristic makes it inherently easier, faster, and more memory-efficient to fine-tune. Users can expedite the fine-tuning process while conserving computational resources by prioritizing adjustments to the mask decoder. This focused approach ensures efficient adaptation of SAM to specific tasks or datasets, optimizing its performance for diverse applications. The steps include: Create a custom dataset by extracting the bounding box coordinates (prompts for the model) and the ground truth segmentation masks. Prepare for fine-tuning by converting the input images to PyTorch tensors, a format SAM's internal functions expect. Run the fine-tuning step by instantiating a training loop that uses the lightweight mask decoder to iterate over the data items, generate masks, and compare them to your ground truth masks. The mask decoder is easier, faster, and more memory-efficient to fine-tune. Compare your tuned model to the original model to make sure there’s indeed a significant improvement. To see the fine-tuning process in practice, check out our detailed blog, How To Fine-Tune Segment Anything, which also includes a Colab notebook as a walkthrough. Real-World Use Cases and Applications SAM can be used in almost every single segmentation task, from instance to panoptic. We’re excited about how quickly SAM can help you pre-label objects with almost pixel-perfect segmentation masks before your expert reviewer adds the ontology on top.  From agriculture and retail to medical imagery and geospatial imagery, the AI-assisted labeling that SAM can achieve is endless. It will be hard to imagine a world where SAM is not a default feature in all major annotation tools. This is why we at Encord are very excited about this new technology. Find other applications that could leverage SAM below.  Image and Video Editors SAM’s outstanding ability to provide accurate segmentation masks for even the most complex videos and images can provide image and video editing applications with automatic object segmentation skills. Whether the prompts (point coordinates, bounding boxes, etc.) are in the foreground or background, SAM uses positional encodings to indicate if the prompt is in the foreground or background. Generating Synthetic Datasets for Low-resource Industries One challenge that has plagued computer vision applications in industries like manufacturing is the lack of datasets. For example, industries building car parts and planning to detect defects in the parts along the production line cannot afford to gather large datasets for that use case. You can use SAM to generate synthetic datasets for your applications. If you realize SAM does not work particularly well for your applications, an option is to fine-tune it on existing datasets. Interested in synthetic data? Read our article, What Is Synthetic Data Generation and Why Is It Useful? Gaze-based Segmentation AR applications can use SAM’s Zero-Shot Single Point Valid Mask Evaluation technique to segment objects through devices like AR glasses based on where subjects gaze. This can help AR technologies give users a more realistic sense of the world as they interact with those objects. Medical Image Segmentation SAM's application in medical image segmentation addresses the demand for accurate and efficient ROI delineation in various medical images. While manual segmentation is accurate, it's time-consuming and labor-intensive. SAM alleviates these challenges by providing semi- or fully automatic segmentation methods, reducing time and labor, and ensuring consistency.  With the adaptation of SAM to medical imaging, MedSAM emerges as the first foundation model for universal medical image segmentation. MedSAM leverages SAM's underlying architecture and training process, providing heightened versatility and potentially more consistent results across different tasks. The MedSAM model consistently does better than the best segmentation foundation models when tested on a wide range of internal and external validation tasks that include different body structures, pathological conditions, and imaging modalities. This underscores MedSAM's potential as a powerful tool for medical image segmentation, offering superior performance compared to specialist models and addressing the critical need for universal models in this domain. MedSAM is open-sourced, and you can find the code for MedSAM on GitHub. Where Does This Leave Us? The Segment Anything Model (SAM) truly represents a groundbreaking development in computer vision. By leveraging promptable segmentation tasks, SAM can adapt to a wide variety of downstream segmentation problems using prompt engineering.  This innovative approach, combined with the largest labeled segmentation dataset to date (SA-1B), allows SAM to achieve state-of-the-art performance in various segmentation tasks. With the potential to significantly enhance AI-assisted labeling and reduce manual labor in image segmentation tasks, SAM can pave the way in industries such as agriculture, retail, medical imagery, and geospatial imagery.  At Encord, we recognize the immense potential of SAM, and we are soon bringing the model to the Encord Platform to support AI-assisted labeling, further streamlining the data annotation process for users. As an open-source model, SAM will inspire further research and development in computer vision, encouraging the AI community to push the boundaries of what is possible in this rapidly evolving field.  Ultimately, SAM marks a new chapter in the story of computer vision, demonstrating the power of foundation models in transforming how we perceive and understand the world around us. Frequently Asked Questions on Segment Anything Model (SAM) How do I fine-tune SAM for my tasks? We have provided a step-by-step walkthrough you can follow to fine-tune SAM for your tasks. Check out the tutorial: How To Fine-Tune Segment Anything. What datasets were used to train SAM? The Segment Anything 1 Billion Mask (SA-1B) dataset has been touted as the “ImageNet of segmentation tasks.” The images vary across subject matter. Scenes, objects, and places frequently appear throughout the dataset. Masks range from large-scale objects such as buildings to fine-grained details like door handles. See the data card and dataset viewer to learn more about the composition of the dataset. Does SAM work well for all tasks? Yes. You can automatically select individual items from images—it works very well on complex images. SAM is a foundation model that provides multi-task capabilities to any application you plug it into. Does SAM work well for ambiguous images? Yes, it does. Because of this, you might find duplicates in your mask sets when you run SAM over your dataset. This allows you to select the most appropriate masks for your task. In this case, you should add a post-processing step to segment the most suitable masks for your task. How long does it take SAM to generate segmentation masks? SAM can generate a segment in as little as 50 milliseconds—practically real-time! Do I need a GPU to run SAM? Although it is possible to run SAM on your CPU, you can use a GPU to achieve significantly faster results from SAM.

Computer vision is having its ChatGPT moment with the release of the Segment Anything Model (SAM) by Meta last week. Trained over 11 billion segmentation masks, SAM is a foundation model for predictive AI use cases rather than generative AI. While it has shown an incredible amount of flexibility in its ability to segment over wide-ranging image modalities and problem spaces, it was released without “fine-tuning” functionality. This tutorial will outline some of the key steps to fine-tune SAM using the mask decoder, particularly describing which functions from SAM to use to pre/post-process the data so that it's in good shape for fine-tuning. What is the Segment Anything Model (SAM)? The Segment Anything Model (SAM) is a segmentation model developed by Meta AI. It is considered the first foundational model for Computer Vision. SAM was trained on a huge corpus of data containing millions of images and billions of masks, making it extremely powerful. As its name suggests, SAM is able to produce accurate segmentation masks for a wide variety of images. SAM’s design allows it to take human prompts into account, making it particularly powerful for Human In The Loop annotation. These prompts can be multi-modal: they can be points on the area to be segmented, a bounding box around the object to be segmented, or a text prompt about what should be segmented. The model is structured into 3 components: an image encoder, a prompt encoder, and a mask decoder. Source The image encoder generates an embedding for the image being segmented, whilst the prompt encoder generates an embedding for the prompts. The image encoder is a particularly large component of the model. This is in contrast to the lightweight mask decoder, which predicts segmentation masks based on the embeddings. Meta AI has made the weights and biases of the model trained on the Segment Anything 1 Billion Mask (SA-1B) dataset available as a model checkpoint. Learn more about how Segment Anything works in our explainer blog post Segment Anything Model (SAM) Explained. What is Model Fine-Tuning? Publicly available state-of-the-art models have a custom architecture and are typically supplied with pre-trained model weights. If these architectures were supplied without weights then the models would need to be trained from scratch by the users, who would need to use massive datasets to obtain state-of-the-art performance. Model fine-tuning is the process of taking a pre-trained model (architecture+weights) and showing it data for a particular use case. This will typically be data that the model hasn’t seen before, or that is underrepresented in its original training dataset. The difference between fine-tuning the model and starting from scratch is the starting value of the weights and biases. If we were training from scratch, these would be randomly initialized according to some strategy. In such a starting configuration, the model would ‘know nothing’ of the task at hand and perform poorly. By using pre-existing weights and biases as a starting point we can ‘fine tune’ the weights and biases so that our model works better on our custom dataset. For example, the information learned to recognize cats (edge detection, counting paws) will be useful for recognizing dogs. Why Would I Fine-Tune a Model? The purpose of fine-tuning a model is to obtain higher performance on data that the pre-trained model has not seen before. For example, an image segmentation model trained on a broad corpus of data gathered from phone cameras will have mostly seen images from a horizontal perspective. If we tried to use this model for satellite imagery taken from a vertical perspective, it may not perform as well. If we were trying to segment rooftops, the model may not yield the best results. The pre-training is useful because the model will have learned how to segment objects in general, so we want to take advantage of this starting point to build a model that can accurately segment rooftops. Furthermore, it is likely that our custom dataset would not have millions of examples, so we want to fine-tune instead of training the model from scratch. Fine tuning is desirable so that we can obtain better performance on our specific use case, without having to incur the computational cost of training a model from scratch. How to Fine-Tune Segment Anything Model [With Code] Background & Architecture We gave an overview of the SAM architecture in the introduction section. The image encoder has a complex architecture with many parameters. In order to fine-tune the model, it makes sense for us to focus on the mask decoder which is lightweight and therefore easier, faster, and more memory efficient to fine-tune. In order to fine-tune SAM, we need to extract the underlying pieces of its architecture (image and prompt encoders, mask decoder). We cannot use SamPredictor.predict (link) for two reasons: We want to fine-tune only the mask decoder This function calls SamPredictor.predict_torch which has the  @torch.no_grad() decorator (link), which prevents us from computing gradients Thus, we need to examine the SamPredictor.predict function and call the appropriate functions with gradient calculation enabled on the part we want to fine-tune (the mask decoder). Doing this is also a good way to learn more about how SAM works. Creating a Custom Dataset We need three things to fine-tune our model: Images on which to draw segmentations Segmentation ground truth masks Prompts to feed into the model We chose the stamp verification dataset (link) since it has data that SAM may not have seen in its training (i.e., stamps on documents). We can verify that it performs well, but not perfectly, on this dataset by running inference with the pre-trained weights. The ground truth masks are also extremely precise, which will allow us to calculate accurate losses. Finally, this dataset contains bounding boxes around the segmentation masks, which we can use as prompts to SAM. An example image is shown below. These bounding boxes align well with the workflow that a human annotator would go through when looking to generate segmentations. Input Data Preprocessing We need to preprocess the scans from numpy arrays to pytorch tensors. To do this, we can follow what happens inside SamPredictor.set_image (link) and SamPredictor.set_torch_image (link) which preprocesses the image. First, we can use utils.transform.ResizeLongestSide to resize the image, as this is the transformer used inside the predictor (link). We can then convert the image to a pytorch tensor and use the SAM preprocess method (link) to finish preprocessing. Training Setup We download the model checkpoint for the vit_b model and load them in: sam_model = sam_model_registry['vit_b'](checkpoint='sam_vit_b_01ec64.pth') We can set up an Adam optimizer with defaults and specify that the parameters to tune are those of the mask decoder: optimizer = torch.optim.Adam(sam_model.mask_decoder.parameters())  At the same time, we can set up our loss function, for example Mean Squared Error loss_fn = torch.nn.MSELoss() Training Loop In the main training loop, we will be iterating through our data items, generating masks, and comparing them to our ground truth masks so that we can optimize the model parameters based on the loss function. In this example, we used a GPU for training since it is much faster than using a CPU. It is important to use .to(device) on the appropriate tensors to make sure that we don’t have certain tensors on the CPU and others on the GPU. We want to embed images by wrapping the encoder in the torch.no_grad() context manager, since otherwise we will have memory issues, along with the fact that we are not looking to fine-tune the image encoder. with torch.no_grad(): image_embedding = sam_model.image_encoder(input_image) We can also generate the prompt embeddings within the no_grad context manager. We use our bounding box coordinates, converted to pytorch tensors. with torch.no_grad(): sparse_embeddings, dense_embeddings = sam_model.prompt_encoder( points=None, boxes=box_torch, masks=None, ) Finally, we can generate the masks. Note that here we are in single mask generation mode (in contrast to the 3 masks that are normally output). low_res_masks, iou_predictions = sam_model.mask_decoder( image_embeddings=image_embedding, image_pe=sam_model.prompt_encoder.get_dense_pe(), sparse_prompt_embeddings=sparse_embeddings, dense_prompt_embeddings=dense_embeddings, multimask_output=False, ) The final step here is to upscale the masks back to the original image size since they are low resolution. We can use Sam.postprocess_masks to achieve this. We will also want to generate binary masks from the predicted masks so that we can compare these to our ground truths. It is important to use torch functionals in order to not break backpropagation. upscaled_masks = sam_model.postprocess_masks(low_res_masks, input_size, original_image_size).to(device) from torch.nn.functional import threshold, normalize binary_mask = normalize(threshold(upscaled_masks, 0.0, 0)).to(device) Finally, we can calculate the loss and run an optimization step: loss = loss_fn(binary_mask, gt_binary_mask) optimizer.zero_grad() loss.backward() optimizer.step() By repeating this over a number of epochs and batches we can fine-tune the SAM decoder. Saving Checkpoints and Starting a Model from it Once we are done with training and satisfied with the performance uplift, we can save the state dict of the tuned model using: torch.save(model.state_dict(), PATH) We can then load this state dict when we want to perform inference on data that is similar to the data we used to fine-tune the model. You can find the Colab Notebook with all the code you need to fine-tune SAM here. Keep reading if you want a fully working solution out of the box! Fine-Tuning for Downstream Applications While SAM does not currently offer fine-tuning out of the box, we are building a custom fine-tuner integrated with the Encord platform. As shown in this post, we fine-tune the decoder in order to achieve this. This is available as an out-of-the-box one-click procedure in the web app, where the hyperparameters are automatically set. Original vanilla SAM mask: Mask generated by fine-tuned version of the model: We can see that this mask is tighter than the original mask. This was the result of fine-tuning on a small subset of images from the stamp verification dataset, and then running the tuned model on a previously unseen example. With further training and more examples, we could obtain even better results. 🔥 NEW RELEASE: We released TTI-Eval (text-to-image evaluation), an open-source library for evaluating zero-shot classification models like CLIP and domain-specific ones like BioCLIP against your (or HF) datasets to estimate how well the model will perform. Get started with it on GitHub, and do ⭐️ the repo if it's awesome. 🔥 Conclusion That's all, folks! You have now learned how to fine-tune the Segment Anything Model (SAM). If you're looking to fine-tune SAM out of the box, you might also be interested to learn that we have recently released the Segment Anything Model in Encord, allowing you to fine-tune the model without writing any code.

Guide to the most popular image annotation tools that you need to know about in 2024. Compare the features and pricing, and choose the best image annotation tool for your use case. It’s 2024—annotating images is still one of the most time-consuming steps in bringing a computer vision project to market. To help you out, we put together a list of the most popular image labeling tools out there. Whether you are: A computer vision team building unmanned drones with your own in-house annotation tool. A team of data scientists working on an autonomous driving project looking for large-scale labeling services. Or a data operations team working in healthcare looking for the right platform for your radiologists to accurately label CT scans. This guide will help you compare the top AI annotation tools and find the right one for you. We will compare each based on key factors - including image annotation service, support for different data types and use cases, QA/QC capabilities, security and data privacy, integration with the machine learning pipeline, and customer support. But first, let's explore the process of selecting an image annotation tool from the available providers. Choosing the right image annotation tool is a critical decision that can significantly impact the quality and efficiency of the annotation process. To make an informed choice, it's essential to consider several factors and evaluate the suitability of an image annotation tool for specific needs. Evaluating Image Annotation Tools for Computer Vision Projects Selecting the perfect image annotation tool is like choosing the perfect brush for your painting. Different projects require specific annotation needs that dictate how downstream components. When evaluating an annotation tool that fits your project specifications, there are a few key factors you have to consider. In this section, we will explore those key factors and practical considerations to help you navigate the selection process and find the most fitting AI annotation tool for your computer vision applications. Annotation Types: An effective labeling tool should support various annotation types, such as bounding boxes (ideal for object localization), polygons (useful for detailed object outlines), keypoints (for pose estimation), and semantic segmentation (for scene understanding). The tool must be adaptable to different annotation requirements, allowing users to annotate images with precision and specificity based on the task at hand. User Interface (UI) and User Experience (UX): The user interface plays a crucial role in the efficiency and accuracy of the annotation process. A good annotation tool should have an intuitive interface that is easy to navigate, reducing the learning curve for users. Clear instructions, user-friendly controls, and efficient workflows contribute to a smoother annotation experience. Scalability: Consider the tool's ability to scale with the growing volume of data. A tool that efficiently handles large datasets and multiple annotators is crucial for projects with evolving requirements. Automation and AI Integration: Look for image labeling tools that offer automation features, such as automatic annotation tools or features, to accelerate the annotation process. Integration with artificial intelligence (AI) algorithms can further enhance efficiency by automating repetitive tasks, reducing manual effort, and improving annotation accuracy. Collaboration and Workflow Management: Assess the data annotation tool's collaboration features, including version control, user roles, and workflow management. Collaboration tools are essential for teams working on complex annotation projects. Data Security and Privacy: Ensure that the tool adheres to data security and privacy standards like GDPR. Evaluate encryption methods, access controls, and policies regarding the handling of sensitive data. Pricing: Consider various pricing models, such as per-user, per-project, or subscription models. Also factor in scalability costs, and potential additional fees, ensuring transparency in the pricing structure. Once you've identified which factors are most important for you to evaluate image annotating tools, the next step is understanding how to assess their suitability for your specific use case.  Most Popular Image Annotation Tools Let's compare the features offered by the best image annotation companies such as Encord, Scale AI, Label Studio, SuperAnnotate, CVAT, and Amazon SageMaker Ground Truth, and understand how they assist in annotating images. This article discusses the top 17 image annotation tools in 2024 to help you choose the right image annotation software for your use case. Encord Scale CVAT Label Studio Labelbox Playment Appen Dataloop SuperAnnotate V7 Labs Hive COCO Annotator Make Sense VGG Image Annotator LabelMe Amazon SageMaker Ground Truth VOTT Encord Encord is an automated annotation platform for AI-assisted image annotation, video annotation, and dataset management.  Key Features Data Management: Compile your raw data into curated datasets, organize datasets into folders, and send datasets for labeling.  AI-assisted Labeling: Automate 97% of your annotations with 99% accuracy using auto-annotation features powered by Meta's Segment Anything Model or GPT-4’s LLaVA. Collaboration: Integrate human-in-the-loop seamlessly with customized Workflows - create workflows with the no-code drag and drop builder to fit your data ops & ML pipelines. Quality Assurance: Robust annotator management & QA workflows to track annotator performance and increase label quality.  Integrated Data Labeling Services for all Industries: outsource your labeling tasks to an expert workforce of vetted, trained and specialized annotators to help you scale. Video Labeling Tool: provides the same support for video annotation. One of the leading video annotation tools with positive customer reviews, providing automated video annotations without frame rate errors. Robust Security Functionality: label audit trails, encryption, FDA, CE Compliance, and HIPAA compliance. Integrations: Advanced Python SDK and API access (+ easy export into JSON and COCO formats). Best for Commercial teams: Teams translating from an in-house solution or open-source tool that require a scalable annotation workflow with a robust, secure, and collaborative enterprise-grade platform. Complex or unique use case: For teams that require advanced annotation tool and functionality. It includes, complex nested ontologies or rendering native DICOM formats. Pricing Simple per-user pricing – no need to track annotation hours, label consumption or data usage.    Curious? Try it out Scale Scale AI, now Scale, is a data and labeling services platform that supports computer vision use cases but specializes in RLHF, user experience optimization, large language models, and synthetic data. Scale AI's Image Annotation Tool Key Features Customizable Workflows: Offers customizable labeling workflows tailored to specific project requirements and use cases. Data labeling services: Provides high-quality data labeling services for various data types, including images, text, audio, and video. Scalability: Capable of handling large-scale annotation projects and accommodating growing datasets and annotation needs. Best for Teams Looking for a Labeling Tool: Scale is a very popular option for data labeling services. Teams Looking for Annotation Tools for Autonomous Vehicle Vision: Scale is one of the earliest platforms on the market to support 3D Sensor Fusion annotation for RADAR and LiDAR use cases. Teams Looking for Medical Imaging Annotation Tools: Platforms like Scale will usually not support DICOM or NIfTI data types nor allow companies to work with their data annotators on the platform. Pricing On a per-image basis CVAT (Computer Vision Annotation Tool) CVAT is an open source image annotation tool that is a web-based annotation toolkit, built by Intel. For image labeling, CVAT supports four types of annotations: points, polygons, bounding boxes, and polylines, as well as a subset of computer vision tasks: image segmentation, object detection, and image classification. In 2022, CVAT’s data, content, and GitHub repository were migrated over to OpenCV, where CVAT continues to be open-source. Furthermore, CVAT can also be utilized to annotate QR codes within images, facilitating the integration of QR code recognition into computer vision pipelines and applications. CVAT Label Editor Key Features Open-source: Easy and free to get started labeling images. Manual Annotation Tools: Supports a wide range of annotation types including bounding boxes, polygons, polylines, points, and cuboids, catering to diverse annotation needs. Multi-platform Compatibility: Works on various operating systems such as Windows, Linux, and macOS, providing flexibility for users. Export Formats: CVAT offers support for various data formats including JSON, COCO, and XML-based like Pascal VOC, ensuring annotation compatibility with diverse tools and platforms. Best for Students, researchers, and academics testing the waters with image annotation (perhaps with a few images or a small dataset). Not preferable for commercial teams as it lacks scalability, collaborative features, and robust security. Pricing Free 💡 More insights on image labeling with CVAT: For a team looking for free image annotation tools, CVAT is one of the most popular open-source tools in the space, with over 1 million downloads since 2021. Other popular free image annotation alternatives to CVAT are 3D Slicer, Labelimg, VoTT (Visual Object Tagging Tool - developed by Microsoft), VIA (VGG Image Annotator), LabelMe, and Label Studio. If data security is a requirement for your annotation project… Commercial labeling tools will most likely be a better fit — key security features like audit trails, encryption, SSO, and generally-required vendor certifications (like SOC2, HIPAA, FDA, and GDPR) are usually not available in open-source tools. Further reading: Overview of open source annotation tools for computer vision Complete guide to image annotation for computer vision    Label Studio Label Studio is another popular open source data labeling platform. It provides a versatile platform for annotating various data types, including images, text, audio, and video. Label Studio supports collaborative labeling, custom labeling interfaces, and integration with machine learning pipelines for data annotation tasks. Label Studio Image Annotation Tool Key Features Customizable Labeling Interfaces: Flexible configuration for tailored annotation interfaces to specific tasks. Collaboration Tools: Real-time annotation and project sharing capabilities for seamless collaboration among annotators. Extensible: Easily connect to cloud object storage and label data there directly Export Formats: Label Studio supports multiple data formats including JSON, CSV, TSV, and VOC XML like Pascal VOC, facilitating integration and annotation from diverse sources for machine learning tasks. Best for Data scientists, machine learning engineers, and researchers or teams requiring versatile data labeling for images.  Not suitable for teams with limited technical expertise or resources for managing an open source tool Price Free with enterprise plan available Labelbox Labelbox is a US-based data annotation platform founded in 2017. Like most of the other platforms mentioned in this guide, Labelbox offers both an image labeling platform, as well as labeling services. Labelbox Image Editor Key Features Data Management: QA workflows and data annotator performance tracking. Customizable Labeling Interface: 3rd party labeling services through Labelbox Boost. Automation: Integration with AI models for automatic data labeling to accelerate the annotation process. Annotation Type: Support for multiple data types beyond images, especially text. Best for Teams looking for a platform to quickly annotate documents and text. Teams carrying out annotation projects that are use-case specific. As generalist tools, platforms like Labelbox are great at handling a broad variety of data types. If you’re working on a unique use-case-specific annotation project (like scans in DICOM formats or high-resolution images that require pixel-perfect annotations), other commercial AI labeling tools will be a better fit: check out our blog exploring Best DICOM Labeling Tools. Pricing Varies based on the volume of data, percent of the total volume needing to be labeled, number of seats, number of projects, and percent of data used in model training. For larger commercial teams, this pricing may get expensive as your project scales. Playment Playment is a fully-managed data annotation platform. The workforce labeling company was acquired by Telus in 2021 and provides computer vision teams with training data for various use cases, supported by manual labelers and a machine learning platform. Playment Image Annotation Tool Key Features Data Labeling Services: Provides high-quality data labeling services for various data types including images, videos, text, and sensor data. Support: Global workforces of contractors and data labelers. Scalability: Capable of handling large-scale annotation projects and accommodating growing datasets and annotation needs. Audio Labeling Tool: Speech recognition training platform (handles all data types across 500+ languages and dialects). Best for Teams looking for a fully managed solution who do not need visibility into the process. Pricing Enterprise plan Appen Appen is a data labeling services platform founded in 1996, making it one of the first and oldest solutions in the market. The company offers data labeling services for a wide range of industries and in 2019, acquired Figure Eight to build out its software capabilities and help businesses also train and improve their computer vision models. Appen Image Annotation Tool Key Features Data Labeling Services: Support for multiple annotation types (bounding boxes, polygons, and image segmentation). Data Collection: Data sourcing (pre-labeled datasets), data preparation, and real-world model evaluation. Natural Language Processing:  Supports natural language processing tasks such as sentiment analysis, entity recognition, and text classification. Image and Video Analysis: Analyzes images and videos for tasks such as object detection, image classification, and video segmentation. Best for Teams looking for image data sourcing and collection alongside annotation services. Pricing Enterprise plan Dataloop Dataloop is an Israel-based data labeling platform that provides a comprehensive solution for data Dataloop is an Israel-based data labeling platform that provides a comprehensive solution for data management and annotation projects. The tool offers data labeling capabilities across images, text, audio, and video annotation, helping businesses train and improve their machine learning models. Dataloop Image Annotation Tool Key Features Data Annotation: Features for image annotation tasks, including classification, detection, and semantic segmentation. Video Annotation Tool: Support for video annotations. Collaboration Tool: Features for real-time collaboration among annotators, project sharing, and version control for efficient teamwork. Data Management: Offers data management capabilities including data versioning, tracking, and organization for streamlined workflows. Best for Teams looking for a generalist annotation tool for various data annotation needs. Teams carrying out specific image and video annotation projects that are use-case specific. As generalist tools, platforms like Dataloop are built to support a wide variety of simple use cases, so other commercial platforms are a better fit if you’re trying to label use-case-specific annotation projects (like high-resolution images that require pixel-perfect annotations in satellite imaging or DICOM files for medical teams). Pricing Free trial and an enterprise plan. SuperAnnotate SuperAnnotate provides enterprise solutions for image and video annotation, catering primarily to the needs of the computer vision community. It provides powerful annotation tools and features tailored for machine learning and AI applications, offering efficient labeling solutions to enhance model training and accuracy. SuperAnnotate - Image Annotation Tool Key Features Multi-Data Type Support: Versatile annotation tool for image, video, text, and audio. AI Assistance: Integrates AI-assisted annotation to accelerate the annotation process and improve efficiency. Customization: Provides customizable annotation interfaces and workflows to tailor annotation tasks according to specific project requirements. Integration: Seamlessly integrates with machine learning pipelines and workflows for efficient model training and deployment. Scalability: Capable of handling large-scale annotation projects and accommodating growing datasets and annotation needs. Export Formats: SuperAnnotate supports multiple data formats, including popular ones like JSON, COCO, and Pascal VOC. Best for Larger teams working on various machine learning solutions looking for a versatile annotation tool. Pricing Free for early stage startups and academics for team size up to 3. Enterprise plan V7 Labs V7 is a UK-based data annotation platform founded in 2018. The company enables teams to annotate training data, support the human-in-the-loop processes, and also connect with annotation services. V7 offers annotation of a wide range of data types alongside image annotation tooling, including documents and videos. V7 Labs Image Annotation Tool Key Features Collaboration Capabilities: Project management and automation workflow functionality, with real-time collaboration and tagging. Data Labeling Services: Provides labeling services for images and videos. AI Assistance: Model-assisted annotation of multiple annotation types (segmentation, detection, and more). Best for Students or teams looking for a generalist platform to easily annotate different data types in one place (like documents, images, and short videos). Limited functionalities for use-case specific annotations. Pricing Various options, including academic, business, and pro. Hive Hive was founded in 2013 and provides cloud-based AI solutions for companies wanting to label content across a wide range of data types, including images, video, audio, text, and more. Hive Image Annotation Tool Key Features Image Annotation Tool: Offers annotation tools and workflows for labeling images along with support for unique image annotation use cases (ad targeting, semi-automated logo detection). Ease of Access: Flexible access to model predictions with a single API call. Integration: Seamlessly integrates with machine learning pipelines and workflows for AI model training and deployment. Best for Teams labeling images and other data types for the purpose of content moderation. Pricing Enterprise plan COCO Annotator COCO Annotator is a web-based image annotation tool, crafted by Justin Brooks under the MIT license. Specifically designed to streamline the process of labeling images for object detection, localization, and keypoints detection models, this tool offers a range of features that cater to the diverse needs of machine learning practitioners and researchers.  COCO Annotator - Image Annotation Tool Key Features Image Annotation: Supports annotation of images for object detection, instance segmentation, keypoint detection, and captioning tasks. Export Formats: To facilitate large-scale object detection, the tool exports and stores annotations in the COCO format.  Automations: The tool makes annotating an image easier by incorporating semi-trained models. Additionally, it provides access to advanced selection tools, including the MaskRCNN, Magic Wand and DEXTR. Best For ML Research Teams: COCO Annotator is a good choice for ML researchers, preferable for image annotation for tasks like object detection and keypoints detection. Price Free Make Sense Make Sense AI is a user-friendly and open-source annotation tool, available under the GPLv3 license. Accessible through a web browser without the need for advanced installations, this tool simplifies the annotation process for various image types. Make Sense - Image Annotation Tool Key Features Open Sourced: Make Sense AI stands out as an open-source tool, freely available under the GPLv3 license, fostering collaboration and community engagement for its ongoing development. Accessibility: It ensures web-based accessibility, operating seamlessly in a web browser without complex installations, promoting ease of use across various devices. Export Formats: It facilitates exporting annotations in multiple formats (YOLO, VOC XML like Pascal VOC, VGG JSON, and CSV), ensuring compatibility with diverse machine learning algorithms and seamless integration into various workflows. Best For Small teams seeking an efficient solution to annotate an image. Price Free VGG Image Annotator VGG Image Annotator (VIA) is a versatile open-source tool crafted by the Visual Geometry Group (VGG) for the manual annotation of both image and video data. Released under the permissive BSD-2 clause license, VIA serves the needs of both academic and commercial users, offering a lightweight and accessible solution for annotation tasks. VGG Image Annotator - Image Annotation Tool Key Features Lightweight and User-Friendly: VIA is a lightweight, self-contained annotation tool, utilizing HTML, Javascript, and CSS without external libraries, enabling offline usage in modern web browsers without setup or installation. Offline Capability: The tool is designed to be used offline, providing a full application experience within a single HTML file of size less than 200 KB.  Multi-User Collaboration: Facilitates collaboration among multiple annotators with features such as project sharing, real-time annotation, and version control. Best For VGG Image Annotator (VIA) is ideal for individuals and small teams involved in projects for academic researchers. Price Free LabelMe LabelMe is an open-source web-based tool developed by the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) that allows users to label and annotate images for computer vision research. It provides a user-friendly interface for drawing bounding boxes, polygons, and semantic segmentation masks to label objects within images. LabelMe Image Annotation Tool Key Features Web-Based: Accessible through a web-based interface, allowing for annotation tasks to be performed in any modern web browser without requiring software installation. Customizable Interface: Provides a customizable annotation interface with options to adjust settings, colors, and layout preferences to suit specific project requirements. Best for Academic and research purposes Pricing Free Amazon SageMaker Ground Truth Amazon SageMaker Ground Truth is a fully managed data labeling service provided by Amazon Web Services (AWS). It offers a platform for efficiently labeling large datasets to train machine learning models. Ground Truth supports various annotation tasks, including image classification, object detection, semantic segmentation, and more. Amazon SageMaker Ground Truth - Image Annotation Tool Key Features Managed Service: Fully managed by AWS, eliminating the need for infrastructure setup and management. Human-in-the-Loop Labeling: Harnesses the power of human feedback across the ML lifecycle to improve the accuracy and relevancy of models. Scalability: Capable of handling large-scale annotation projects and accommodating growing datasets and annotation needs. Integration with Amazon SageMaker: Seamlessly integrates with Amazon SageMaker for model training and deployment, providing a streamlined end-to-end machine learning workflow. Best for Teams requiring large-scale data labeling. Pricing Varies based on labeling task and type of data. VOTT VOTT or Visual Object Tagging Tool is an open-source tool developed by Microsoft for annotating images and videos to create training datasets for computer vision models. VOTT provides an intuitive interface for drawing bounding boxes around objects of interest and labeling them with corresponding class names. VOTT Image Annotation Tool Key Features Versatile Annotation Tool: Supports a wide range of annotation types including bounding boxes, polygons, polylines, points, and segmentation masks for precise labeling. Video Annotation: Enables annotation of videos frame by frame, with support for object tracking and interpolation to streamline the annotation process. Multi-Platform Compatibility: Works across various operating systems such as Windows, Linux, and macOS, ensuring flexibility for users. Best for Teams requiring lightweight and customizable annotation tool for object detection. Pricing Free Image Annotation Tool: Key Takeaways There you have it! The 17 Best Image Annotation Tools for computer vision in 2024.  For further reading, you might also want to check out a few 2024 honorable mentions, both paid and free annotation tools: Supervisely - commercial data labeling platform praised for its quality control functionality and basic interpolation feature. Labelimg - Labelimg is an open source multi-modal data annotation tool now part of Label Studio. MarkUp - MarkUp image is a free web annotation tool to annotate an image or a PDF.