Introduction to Balanced and Imbalanced Datasets in Machine Learning

The train-validation-test split is fundamental to the development of robust and reliable machine learning models. Put very simply: The goal: Create a neural network that generalizes well to new data The motto: Never train that model on test data The premise is intuitive – avoid employing the same dataset to train your machine learning model that you use to evaluate it. Doing so will result in a biased model that reports an artificially high model accuracy against the dataset it was trained on and poor model accuracy against any other dataset.  To ensure the generalizability of your machine learning algorithm, it is crucial to split the dataset into three segments: the training set, validation set, and test set. This will allow you to realistically measure your model’s performance by ensuring that the dataset used to train the model and the dataset used to evaluate it are distinct.  In this article, we outline: Training Set vs. Validation Set vs. Test Sets 3 Methods to Split Machine Learning Datasets 3 Mistakes to Avoid When Splitting Machine Learning Datasets  How to use Encord for Training, Validation, and Test Splits in Computer Vision Key Takeaways Training vs. Validation vs. Test Sets Before we dive into the best practices of the train-validation-test split for machine learning models, let’s define the three sets of data. Training Set The training set is the portion of the dataset reserved to fit the model. In other words, the model sees and learns from the data in the training set to directly improve its parameters.  To maximize model performance, the training set must be (i) large enough to yield meaningful results (but not too large that the model overfits) and (ii) representative of the dataset as a whole. This will allow the trained model to predict any unseen data that may appear in the future. Overfitting occurs when a machine learning model is too specialized and adapted to the training data that it is unable to generalize and make correct predictions on new data. As a result, an overfit model will overperform with the training set but underperform when presented with validation sets and test sets.  💡 Encord pioneered the micro model methodology, which capitalizes on overfitting machine learning models on narrowly defined tasks to be applied across entire datasets once model accuracy is high. Learn more about micro models in An Introduction to Micro-Models for Labeling Images and Videos Validation Set The validation set is the set of data used to evaluate and fine-tune a machine learning model during training, helping to assess the model’s performance and make adjustments.  By evaluating a trained model on the validation set, we gain insights into its ability to generalize to unseen data. This assessment helps identify potential issues such as overfitting, which can have a significant impact on the model’s performance in real-world scenarios. The validation set is also essential for hyperparameter tuning. Hyperparameters are settings that control the behavior of the model, such as learning rate or regularization strength. By experimenting with different hyperparameter values, training the model on the training set, and evaluating its performance with the validation set, we can identify the optimal combination of hyperparameters that yields the best results. This iterative process fine-tunes the model and maximizes its performance. Test Set The test set is the set of data used to evaluate the final performance of a trained model.  It serves as an unbiased measure of how well the model generalizes to unseen data, assessing its generalization capabilities in real-world scenarios. By keeping the test set separate throughout the development process, we obtain a reliable benchmark of the model’s performance.  The test dataset also helps gauge the trained model's ability to handle new data. Since it represents unseen data that the model has never encountered before, evaluating the model fit on the test set provides an unbiased metric into its practical applicability. This assessment enables us to determine if the trained model has successfully learned relevant patterns and can make accurate predictions beyond the training and validation contexts. 3 Methods to Split Machine Learning Datasets There are various methods of splitting datasets for machine learning models. The right approach for data splitting and the optimal split ratio both depend on several factors, including the use case, amount of data, quality of data, and the number of hyperparameters.  Random Sampling The most common approach for dividing a dataset is random sampling. As the name suggests, the method involves shuffling the dataset and randomly assigning samples to training, validation, or test sets according to predetermined ratios. With class-balanced datasets, random sampling ensures the split is unbiased.  While random sampling is the best approach for many ML problems, it is not the correct approach with imbalanced datasets. When the data consists of skewed class proportions, random sampling will almost certainly create a bias in the model. Stratified Dataset Splitting Stratified dataset splitting is a method commonly used with imbalanced datasets, where certain classes or categories have significantly fewer instances than others. In such cases, it is crucial to ensure that the training, validation, and test sets adequately represent the class distribution to avoid bias in the final model. In stratified splitting, the dataset is divided while preserving the relative proportions of each class across the splits. As a result, the training, validation, and test sets contain a representative subset from each class, maintaining the original class distribution. By doing so, the model can learn to recognize patterns and make predictions for all classes, resulting in a more robust and reliable machine learning algorithm. To learn more, read Introduction to Balanced and Imbalanced Datasets in Machine Learning. Cross-Validation Splitting Cross-validation sampling is a technique used to split a dataset into training and validation sets for cross-validation purposes. It involves creating multiple subsets of the data, each serving as a training set or validation set during different iterations of the cross-validation process.  K-fold cross-validation and stratified k-fold cross-validation are common techniques. By utilizing these cross-validation sampling techniques, researchers and machine learning practitioners can obtain more reliable and unbiased performance metrics for their machine learning models, enabling them to make better-informed decisions during model development and selection. 3 Mistakes to Avoid When Data Splitting  There are some common pitfalls that data scientists and ML engineers make when splitting datasets for model training.  Inadequate Sample Size Insufficient sample size in the training, validation, or test sets can lead to unreliable model performance metrics. If the training set is too small, the model may not capture enough patterns or generalize well. Similarly, if the validation set or test set is too small, the performance evaluation may lack statistical significance.  Data Leakage Data leakage occurs when information from the validation set or test set inadvertently leaks into the training set. This can lead to overly optimistic performance metrics and an inflated sense of the final model accuracy. To prevent data leakage, it is crucial to ensure strict separation between the training set, validation set, and test set, making sure that no information from the evaluation sets are used during model training. Improper Shuffle or Sorting Incorrectly shuffling or sorting the data before splitting can introduce bias and affect the generalization of the final model. For example, if the dataset is not shuffled randomly before splitting into training set and validation set, it may introduce biases or patterns that the model can exploit during training. As a result, the trained model may overfit to those specific patterns and fail to generalize well to new, unseen data. How to use Encord for Training, Validation, and Test Splits To get started with a machine learning project, Encord’s platform can be used for data splitting. To download Encord Active, run the following commands in your preferred Python environment: python3.9 -m venv ea-venv source ea-venv/bin/activate # within venv pip install encord-active Alternatively, run the following command to install Encord Active using GitHub: pip install git+https://github.com/encord-team/encord-active To confirm Encord Active has been installed, run the following: encord-active --help Encord Active has many sandbox datasets like the COCO, Rareplanes, BDD100K, TACO dataset and many more. These sandbox datasets are commonly used in computer vision applications for building benchmark models. Now that Encord Active has been installed, download the COCO dataset with the following command: encord-active download The script prompts you to choose a project, navigate the options ↓ and ↑ select the COCO dataset, and hit enter. The COCO dataset referred to here is the same COCO validation dataset mentioned in the COCO webpage. This dataset is used to demonstrate how Encord Active can filter and split the dataset easily. In order to visualize the data in the browser, run the following command: cd /path/to/downloaded/project encord-active visualize The image displayed below is the webpage that opens in your browser, showcasing the data and its properties. Let’s examine the properties: Visualize the data in your browser (data = COCO dataset) The COCO dataset comprises 5000 images, of which we have allocated 3500 images for the training set, 1000 images for the validation set, and 500 images for the test set. The filter can be used to select images based on their features, ensuring the subsets of data are balanced. As illustrated below, in Data Quality→Explorer, use the features to filter out images. Filtering the dataset based on Blur, Area, Brightness, and Frame object density values for the training set Example of images in the training set Through the Action tab in the Toolbox, click on Create Subset to create a new training subset. You can either download the subset in CSV or COCO format, or use it in the Encord Platform to train and evaluate your machine learning model. Similar to the training subset, create a validation set and test set. Remember to choose the same features but filter by different values to avoid data leakage. Check out our curated lists of open-source datasets for various sectors, including Top 10 Free Healthcare Datasets for Computer Vision, Top 8 Free Datasets for Human Pose Estimation in Computer Vision, and Best Datasets for Computer Vision | Industry Breakdown. Training, Validation, and Test Set: Key Takeaways Here are the key takeaways: In order to create a model that generalizes well to new data, it is important to split data into training, validation, and test sets to prevent evaluating the model on the same data used to train it. The training set data must be large enough to capture variability in the data but not so large that the model overfits the training data.  The optimal split ratio depends on various factors. The rough standard for train-validation-test splits is 60-80% training data, 10-20% validation data, and 10-20% test data. Frequently Asked Questions What is the train-test split? The train-test split is a technique in machine learning where a dataset is divided into two subsets: the training set and test set. The training set is used to train the model, while the test set is used to evaluate the final model’s performance and generalization capabilities.  What is the purpose of the validation set? The validation set provides an unbiased assessment of the model’s performance on unseen data before finalizing. It helps in fine-tuning the model’s hyperparameters, selecting the best model, and preventing overfitting.  Why is it important to split your data? It is important to split your data into a training set and test set to evaluate the model performance and generalizability ability of a machine learning algorithm. By using separate sets, you can simulate the model’s performance on unseen data, detect overfitting, optimize model parameters, and make informed decisions about its effectiveness before deployment. What is overfitting? Overfitting occurs when a machine learning model performs well on the training data but fails to generalize to new, unseen data. This occurs when the trained model learns noise or irrelevant patterns from the training set, leading to poor model performance on the test set or validation set. What is cross-validation? Cross-validation is a technique used to evaluate the model performance and generalization capabilities of a machine learning algorithm. It involves dividing the dataset into multiple subsets or folds. The machine learning model is trained on a combination of these subsets while being tested on the remaining subset. This process is repeated, and performance metrics are averaged to assess model performance. Recommended Articles How to Build Your First Machine Learning Model The Full Guide to Training Datasets for Machine Learning Active Learning in Machine Learning [Guide & Strategies]    Self-Supervised Learning Explained DINOv2: Self-supervised Learning Model Explained 4 Questions to Ask When Evaluating Training Data Pipelines How to Automate the Assessment of Training Data Quality 5 Ways to Reduce Bias in Computer Vision Datasets 9 Ways to Balance Your Computer Vision Dataset How to Improve Datasets for Computer Vision    How to Automate Data Labeling [Examples + Tutorial] The Complete Guide to Object Tracking [Tutorial]

Computer vision is having its ChatGPT moment with the release of the Segment Anything Model (SAM) by Meta last week. Trained over 11 billion segmentation masks, SAM is a foundation model for predictive AI use cases rather than generative AI. While it has shown an incredible amount of flexibility in its ability to segment over wide-ranging image modalities and problem spaces, it was released without “fine-tuning” functionality. This tutorial will outline some of the key steps to fine-tune SAM using the mask decoder, particularly describing which functions from SAM to use to pre/post-process the data so that it's in good shape for fine-tuning. What is the Segment Anything Model (SAM)? The Segment Anything Model (SAM) is a segmentation model developed by Meta AI. It is considered the first foundational model for Computer Vision. SAM was trained on a huge corpus of data containing millions of images and billions of masks, making it extremely powerful. As its name suggests, SAM is able to produce accurate segmentation masks for a wide variety of images. SAM’s design allows it to take human prompts into account, making it particularly powerful for Human In The Loop annotation. These prompts can be multi-modal: they can be points on the area to be segmented, a bounding box around the object to be segmented, or a text prompt about what should be segmented. The model is structured into 3 components: an image encoder, a prompt encoder, and a mask decoder. Source The image encoder generates an embedding for the image being segmented, whilst the prompt encoder generates an embedding for the prompts. The image encoder is a particularly large component of the model. This is in contrast to the lightweight mask decoder, which predicts segmentation masks based on the embeddings. Meta AI has made the weights and biases of the model trained on the Segment Anything 1 Billion Mask (SA-1B) dataset available as a model checkpoint. Learn more about how Segment Anything works in our explainer blog post Segment Anything Model (SAM) Explained. What is Model Fine-Tuning? Publicly available state-of-the-art models have a custom architecture and are typically supplied with pre-trained model weights. If these architectures were supplied without weights then the models would need to be trained from scratch by the users, who would need to use massive datasets to obtain state-of-the-art performance. Model fine-tuning is the process of taking a pre-trained model (architecture+weights) and showing it data for a particular use case. This will typically be data that the model hasn’t seen before, or that is underrepresented in its original training dataset. The difference between fine-tuning the model and starting from scratch is the starting value of the weights and biases. If we were training from scratch, these would be randomly initialized according to some strategy. In such a starting configuration, the model would ‘know nothing’ of the task at hand and perform poorly. By using pre-existing weights and biases as a starting point we can ‘fine tune’ the weights and biases so that our model works better on our custom dataset. For example, the information learned to recognize cats (edge detection, counting paws) will be useful for recognizing dogs. Why Would I Fine-Tune a Model? The purpose of fine-tuning a model is to obtain higher performance on data that the pre-trained model has not seen before. For example, an image segmentation model trained on a broad corpus of data gathered from phone cameras will have mostly seen images from a horizontal perspective. If we tried to use this model for satellite imagery taken from a vertical perspective, it may not perform as well. If we were trying to segment rooftops, the model may not yield the best results. The pre-training is useful because the model will have learned how to segment objects in general, so we want to take advantage of this starting point to build a model that can accurately segment rooftops. Furthermore, it is likely that our custom dataset would not have millions of examples, so we want to fine-tune instead of training the model from scratch. Fine tuning is desirable so that we can obtain better performance on our specific use case, without having to incur the computational cost of training a model from scratch. How to Fine-Tune Segment Anything Model [With Code] Background & Architecture We gave an overview of the SAM architecture in the introduction section. The image encoder has a complex architecture with many parameters. In order to fine-tune the model, it makes sense for us to focus on the mask decoder which is lightweight and therefore easier, faster, and more memory efficient to fine-tune. In order to fine-tune SAM, we need to extract the underlying pieces of its architecture (image and prompt encoders, mask decoder). We cannot use SamPredictor.predict (link) for two reasons: We want to fine-tune only the mask decoder This function calls SamPredictor.predict_torch which has the  @torch.no_grad() decorator (link), which prevents us from computing gradients Thus, we need to examine the SamPredictor.predict function and call the appropriate functions with gradient calculation enabled on the part we want to fine-tune (the mask decoder). Doing this is also a good way to learn more about how SAM works. Creating a Custom Dataset We need three things to fine-tune our model: Images on which to draw segmentations Segmentation ground truth masks Prompts to feed into the model We chose the stamp verification dataset (link) since it has data that SAM may not have seen in its training (i.e., stamps on documents). We can verify that it performs well, but not perfectly, on this dataset by running inference with the pre-trained weights. The ground truth masks are also extremely precise, which will allow us to calculate accurate losses. Finally, this dataset contains bounding boxes around the segmentation masks, which we can use as prompts to SAM. An example image is shown below. These bounding boxes align well with the workflow that a human annotator would go through when looking to generate segmentations. Input Data Preprocessing We need to preprocess the scans from numpy arrays to pytorch tensors. To do this, we can follow what happens inside SamPredictor.set_image (link) and SamPredictor.set_torch_image (link) which preprocesses the image. First, we can use utils.transform.ResizeLongestSide to resize the image, as this is the transformer used inside the predictor (link). We can then convert the image to a pytorch tensor and use the SAM preprocess method (link) to finish preprocessing. Training Setup We download the model checkpoint for the vit_b model and load them in: sam_model = sam_model_registry['vit_b'](checkpoint='sam_vit_b_01ec64.pth') We can set up an Adam optimizer with defaults and specify that the parameters to tune are those of the mask decoder: optimizer = torch.optim.Adam(sam_model.mask_decoder.parameters())  At the same time, we can set up our loss function, for example Mean Squared Error loss_fn = torch.nn.MSELoss() Training Loop In the main training loop, we will be iterating through our data items, generating masks, and comparing them to our ground truth masks so that we can optimize the model parameters based on the loss function. In this example, we used a GPU for training since it is much faster than using a CPU. It is important to use .to(device) on the appropriate tensors to make sure that we don’t have certain tensors on the CPU and others on the GPU. We want to embed images by wrapping the encoder in the torch.no_grad() context manager, since otherwise we will have memory issues, along with the fact that we are not looking to fine-tune the image encoder. with torch.no_grad(): image_embedding = sam_model.image_encoder(input_image) We can also generate the prompt embeddings within the no_grad context manager. We use our bounding box coordinates, converted to pytorch tensors. with torch.no_grad(): sparse_embeddings, dense_embeddings = sam_model.prompt_encoder( points=None, boxes=box_torch, masks=None, ) Finally, we can generate the masks. Note that here we are in single mask generation mode (in contrast to the 3 masks that are normally output). low_res_masks, iou_predictions = sam_model.mask_decoder( image_embeddings=image_embedding, image_pe=sam_model.prompt_encoder.get_dense_pe(), sparse_prompt_embeddings=sparse_embeddings, dense_prompt_embeddings=dense_embeddings, multimask_output=False, ) The final step here is to upscale the masks back to the original image size since they are low resolution. We can use Sam.postprocess_masks to achieve this. We will also want to generate binary masks from the predicted masks so that we can compare these to our ground truths. It is important to use torch functionals in order to not break backpropagation. upscaled_masks = sam_model.postprocess_masks(low_res_masks, input_size, original_image_size).to(device) from torch.nn.functional import threshold, normalize binary_mask = normalize(threshold(upscaled_masks, 0.0, 0)).to(device) Finally, we can calculate the loss and run an optimization step: loss = loss_fn(binary_mask, gt_binary_mask) optimizer.zero_grad() loss.backward() optimizer.step() By repeating this over a number of epochs and batches we can fine-tune the SAM decoder. Saving Checkpoints and Starting a Model from it Once we are done with training and satisfied with the performance uplift, we can save the state dict of the tuned model using: torch.save(model.state_dict(), PATH) We can then load this state dict when we want to perform inference on data that is similar to the data we used to fine-tune the model. You can find the Colab Notebook with all the code you need to fine-tune SAM here. Keep reading if you want a fully working solution out of the box! Fine-Tuning for Downstream Applications While SAM does not currently offer fine-tuning out of the box, we are building a custom fine-tuner integrated with the Encord platform. As shown in this post, we fine-tune the decoder in order to achieve this. This is available as an out-of-the-box one-click procedure in the web app, where the hyperparameters are automatically set. Original vanilla SAM mask: Mask generated by fine-tuned version of the model: We can see that this mask is tighter than the original mask. This was the result of fine-tuning on a small subset of images from the stamp verification dataset, and then running the tuned model on a previously unseen example. With further training and more examples, we could obtain even better results. Conclusion That's all, folks! You have now learned how to fine-tune the Segment Anything Model (SAM). If you're looking to fine-tune SAM out of the box, you might also be interested to learn that we have recently released the Segment Anything Model in Encord, allowing you to fine-tune the model without writing any code.

Guide to the most popular image annotation tools that you need to know about in 2024. Compare the features and pricing, and choose the best image annotation tool for your use case. It’s 2024—annotating images is still one of the most time-consuming steps in bringing a computer vision project to market. To help you out, we put together a list of the most popular image labeling tools out there. Whether you are: A computer vision team building unmanned drones with your own in-house annotation tool. A team of data scientists working on an autonomous driving project looking for large-scale labeling services. Or a data operations team working in healthcare looking for the right platform for your radiologists to accurately label CT scans. This guide will help you compare the top AI annotation tools and find the right one for you. We will compare each based on key factors - including image annotation service, support for different data types and use cases, QA/QC capabilities, security and data privacy, integration with the machine learning pipeline, and customer support. But first, let's explore the process of selecting an image annotation tool from the available providers. Choosing the right image annotation tool is a critical decision that can significantly impact the quality and efficiency of the annotation process. To make an informed choice, it's essential to consider several factors and evaluate the suitability of an image annotation tool for specific needs. Evaluating Image Annotation Tools for Computer Vision Projects Selecting the perfect image annotation tool is like choosing the perfect brush for your painting. Different projects require specific annotation needs that dictate how downstream components. When evaluating an annotation tool that fits your project specifications, there are a few key factors you have to consider. In this section, we will explore those key factors and practical considerations to help you navigate the selection process and find the most fitting AI annotation tool for your computer vision applications. Annotation Types: An effective labeling tool should support various annotation types, such as bounding boxes (ideal for object localization), polygons (useful for detailed object outlines), keypoints (for pose estimation), and semantic segmentation (for scene understanding). The tool must be adaptable to different annotation requirements, allowing users to annotate images with precision and specificity based on the task at hand. User Interface (UI) and User Experience (UX): The user interface plays a crucial role in the efficiency and accuracy of the annotation process. A good annotation tool should have an intuitive interface that is easy to navigate, reducing the learning curve for users. Clear instructions, user-friendly controls, and efficient workflows contribute to a smoother annotation experience. Scalability: Consider the tool's ability to scale with the growing volume of data. A tool that efficiently handles large datasets and multiple annotators is crucial for projects with evolving requirements. Automation and AI Integration: Look for image labeling tools that offer automation features, such as automatic annotation tools or features, to accelerate the annotation process. Integration with artificial intelligence (AI) algorithms can further enhance efficiency by automating repetitive tasks, reducing manual effort, and improving annotation accuracy. Collaboration and Workflow Management: Assess the data annotation tool's collaboration features, including version control, user roles, and workflow management. Collaboration tools are essential for teams working on complex annotation projects. Data Security and Privacy: Ensure that the tool adheres to data security and privacy standards like GDPR. Evaluate encryption methods, access controls, and policies regarding the handling of sensitive data. Pricing: Consider various pricing models, such as per-user, per-project, or subscription models. Also factor in scalability costs, and potential additional fees, ensuring transparency in the pricing structure. Once you've identified which factors are most important for you to evaluate image annotating tools, the next step is understanding how to assess their suitability for your specific use case.  Most Popular Image Annotation Tools Let's compare the features offered by the best image annotation companies such as Encord, Scale AI, Label Studio, SuperAnnotate, CVAT, and Amazon SageMaker Ground Truth, and understand how they assist in annotating images. This article discusses the top 17 image annotation tools in 2024 to help you choose the right image annotation software for your use case. Encord Scale CVAT Label Studio Labelbox Playment Appen Dataloop SuperAnnotate V7 Labs Hive COCO Annotator Make Sense VGG Image Annotator LabelMe Amazon SageMaker Ground Truth VOTT Encord Encord is an automated annotation platform for AI-assisted image annotation, video annotation, and dataset management.  Key Features Data Management: Compile your raw data into curated datasets, organize datasets into folders, and send datasets for labeling.  AI-assisted Labeling: Automate 97% of your annotations with 99% accuracy using auto-annotation features powered by Meta's Segment Anything Model or GPT-4’s LLaVA. Collaboration: Integrate human-in-the-loop seamlessly with customized Workflows - create workflows with the no-code drag and drop builder to fit your data ops & ML pipelines. Quality Assurance: Robust annotator management & QA workflows to track annotator performance and increase label quality.  Integrated Data Labeling Services for all Industries: outsource your labeling tasks to an expert workforce of vetted, trained and specialized annotators to help you scale. Video Labeling Tool: provides the same support for video annotation. One of the leading video annotation tools with positive customer reviews, providing automated video annotations without frame rate errors. Robust Security Functionality: label audit trails, encryption, FDA, CE Compliance, and HIPAA compliance. Integrations: Advanced Python SDK and API access (+ easy export into JSON and COCO formats). Best for Commercial teams: Teams translating from an in-house solution or open-source tool that require a scalable annotation workflow with a robust, secure, and collaborative enterprise-grade platform. Complex or unique use case: For teams that require advanced annotation tool and functionality. It includes, complex nested ontologies or rendering native DICOM formats. Pricing Simple per-user pricing – no need to track annotation hours, label consumption or data usage.    Curious? Try it out Scale Scale AI, now Scale, is a data and labeling services platform that supports computer vision use cases but specializes in RLHF, user experience optimization, large language models, and synthetic data. Scale AI's Image Annotation Tool Key Features Customizable Workflows: Offers customizable labeling workflows tailored to specific project requirements and use cases. Data labeling services: Provides high-quality data labeling services for various data types, including images, text, audio, and video. Scalability: Capable of handling large-scale annotation projects and accommodating growing datasets and annotation needs. Best for Teams Looking for a Labeling Tool: Scale is a very popular option for data labeling services. Teams Looking for Annotation Tools for Autonomous Vehicle Vision: Scale is one of the earliest platforms on the market to support 3D Sensor Fusion annotation for RADAR and LiDAR use cases. Teams Looking for Medical Imaging Annotation Tools: Platforms like Scale will usually not support DICOM or NIfTI data types nor allow companies to work with their data annotators on the platform. Pricing On a per-image basis CVAT (Computer Vision Annotation Tool) CVAT is an open source image annotation tool that is a web-based annotation toolkit, built by Intel. For image labeling, CVAT supports four types of annotations: points, polygons, bounding boxes, and polylines, as well as a subset of computer vision tasks: image segmentation, object detection, and image classification. In 2022, CVAT’s data, content, and GitHub repository were migrated over to OpenCV, where CVAT continues to be open-source. Furthermore, CVAT can also be utilized to annotate QR codes within images, facilitating the integration of QR code recognition into computer vision pipelines and applications. CVAT Label Editor Key Features Open-source: Easy and free to get started labeling images. Manual Annotation Tools: Supports a wide range of annotation types including bounding boxes, polygons, polylines, points, and cuboids, catering to diverse annotation needs. Multi-platform Compatibility: Works on various operating systems such as Windows, Linux, and macOS, providing flexibility for users. Export Formats: CVAT offers support for various data formats including JSON, COCO, and XML-based like Pascal VOC, ensuring annotation compatibility with diverse tools and platforms. Best for Students, researchers, and academics testing the waters with image annotation (perhaps with a few images or a small dataset). Not preferable for commercial teams as it lacks scalability, collaborative features, and robust security. Pricing Free 💡 More insights on image labeling with CVAT: For a team looking for free image annotation tools, CVAT is one of the most popular open-source tools in the space, with over 1 million downloads since 2021. Other popular free image annotation alternatives to CVAT are 3D Slicer, Labelimg, VoTT (Visual Object Tagging Tool - developed by Microsoft), VIA (VGG Image Annotator), LabelMe, and Label Studio. If data security is a requirement for your annotation project… Commercial labeling tools will most likely be a better fit — key security features like audit trails, encryption, SSO, and generally-required vendor certifications (like SOC2, HIPAA, FDA, and GDPR) are usually not available in open-source tools. Further reading: Overview of open source annotation tools for computer vision Complete guide to image annotation for computer vision    Label Studio Label Studio is another popular open source data labeling platform. It provides a versatile platform for annotating various data types, including images, text, audio, and video. Label Studio supports collaborative labeling, custom labeling interfaces, and integration with machine learning pipelines for data annotation tasks. Label Studio Image Annotation Tool Key Features Customizable Labeling Interfaces: Flexible configuration for tailored annotation interfaces to specific tasks. Collaboration Tools: Real-time annotation and project sharing capabilities for seamless collaboration among annotators. Extensible: Easily connect to cloud object storage and label data there directly Export Formats: Label Studio supports multiple data formats including JSON, CSV, TSV, and VOC XML like Pascal VOC, facilitating integration and annotation from diverse sources for machine learning tasks. Best for Data scientists, machine learning engineers, and researchers or teams requiring versatile data labeling for images.  Not suitable for teams with limited technical expertise or resources for managing an open source tool Price Free with enterprise plan available Labelbox Labelbox is a US-based data annotation platform founded in 2017. Like most of the other platforms mentioned in this guide, Labelbox offers both an image labeling platform, as well as labeling services. Labelbox Image Editor Key Features Data Management: QA workflows and data annotator performance tracking. Customizable Labeling Interface: 3rd party labeling services through Labelbox Boost. Automation: Integration with AI models for automatic data labeling to accelerate the annotation process. Annotation Type: Support for multiple data types beyond images, especially text. Best for Teams looking for a platform to quickly annotate documents and text. Teams carrying out annotation projects that are use-case specific. As generalist tools, platforms like Labelbox are great at handling a broad variety of data types. If you’re working on a unique use-case-specific annotation project (like scans in DICOM formats or high-resolution images that require pixel-perfect annotations), other commercial AI labeling tools will be a better fit: check out our blog exploring Best DICOM Labeling Tools. Pricing Varies based on the volume of data, percent of the total volume needing to be labeled, number of seats, number of projects, and percent of data used in model training. For larger commercial teams, this pricing may get expensive as your project scales. Playment Playment is a fully-managed data annotation platform. The workforce labeling company was acquired by Telus in 2021 and provides computer vision teams with training data for various use cases, supported by manual labelers and a machine learning platform. Playment Image Annotation Tool Key Features Data Labeling Services: Provides high-quality data labeling services for various data types including images, videos, text, and sensor data. Support: Global workforces of contractors and data labelers. Scalability: Capable of handling large-scale annotation projects and accommodating growing datasets and annotation needs. Audio Labeling Tool: Speech recognition training platform (handles all data types across 500+ languages and dialects). Best for Teams looking for a fully managed solution who do not need visibility into the process. Pricing Enterprise plan Appen Appen is a data labeling services platform founded in 1996, making it one of the first and oldest solutions in the market. The company offers data labeling services for a wide range of industries and in 2019, acquired Figure Eight to build out its software capabilities and help businesses also train and improve their computer vision models. Appen Image Annotation Tool Key Features Data Labeling Services: Support for multiple annotation types (bounding boxes, polygons, and image segmentation). Data Collection: Data sourcing (pre-labeled datasets), data preparation, and real-world model evaluation. Natural Language Processing:  Supports natural language processing tasks such as sentiment analysis, entity recognition, and text classification. Image and Video Analysis: Analyzes images and videos for tasks such as object detection, image classification, and video segmentation. Best for Teams looking for image data sourcing and collection alongside annotation services. Pricing Enterprise plan Dataloop Dataloop is an Israel-based data labeling platform that provides a comprehensive solution for data Dataloop is an Israel-based data labeling platform that provides a comprehensive solution for data management and annotation projects. The tool offers data labeling capabilities across images, text, audio, and video annotation, helping businesses train and improve their machine learning models. Dataloop Image Annotation Tool Key Features Data Annotation: Features for image annotation tasks, including classification, detection, and semantic segmentation. Video Annotation Tool: Support for video annotations. Collaboration Tool: Features for real-time collaboration among annotators, project sharing, and version control for efficient teamwork. Data Management: Offers data management capabilities including data versioning, tracking, and organization for streamlined workflows. Best for Teams looking for a generalist annotation tool for various data annotation needs. Teams carrying out specific image and video annotation projects that are use-case specific. As generalist tools, platforms like Dataloop are built to support a wide variety of simple use cases, so other commercial platforms are a better fit if you’re trying to label use-case-specific annotation projects (like high-resolution images that require pixel-perfect annotations in satellite imaging or DICOM files for medical teams). Pricing Free trial and an enterprise plan. SuperAnnotate SuperAnnotate provides enterprise solutions for image and video annotation, catering primarily to the needs of the computer vision community. It provides powerful annotation tools and features tailored for machine learning and AI applications, offering efficient labeling solutions to enhance model training and accuracy. SuperAnnotate - Image Annotation Tool Key Features Multi-Data Type Support: Versatile annotation tool for image, video, text, and audio. AI Assistance: Integrates AI-assisted annotation to accelerate the annotation process and improve efficiency. Customization: Provides customizable annotation interfaces and workflows to tailor annotation tasks according to specific project requirements. Integration: Seamlessly integrates with machine learning pipelines and workflows for efficient model training and deployment. Scalability: Capable of handling large-scale annotation projects and accommodating growing datasets and annotation needs. Export Formats: SuperAnnotate supports multiple data formats, including popular ones like JSON, COCO, and Pascal VOC. Best for Larger teams working on various machine learning solutions looking for a versatile annotation tool. Pricing Free for early stage startups and academics for team size up to 3. Enterprise plan V7 Labs V7 is a UK-based data annotation platform founded in 2018. The company enables teams to annotate training data, support the human-in-the-loop processes, and also connect with annotation services. V7 offers annotation of a wide range of data types alongside image annotation tooling, including documents and videos. V7 Labs Image Annotation Tool Key Features Collaboration Capabilities: Project management and automation workflow functionality, with real-time collaboration and tagging. Data Labeling Services: Provides labeling services for images and videos. AI Assistance: Model-assisted annotation of multiple annotation types (segmentation, detection, and more). Best for Students or teams looking for a generalist platform to easily annotate different data types in one place (like documents, images, and short videos). Limited functionalities for use-case specific annotations. Pricing Various options, including academic, business, and pro. Hive Hive was founded in 2013 and provides cloud-based AI solutions for companies wanting to label content across a wide range of data types, including images, video, audio, text, and more. Hive Image Annotation Tool Key Features Image Annotation Tool: Offers annotation tools and workflows for labeling images along with support for unique image annotation use cases (ad targeting, semi-automated logo detection). Ease of Access: Flexible access to model predictions with a single API call. Integration: Seamlessly integrates with machine learning pipelines and workflows for AI model training and deployment. Best for Teams labeling images and other data types for the purpose of content moderation. Pricing Enterprise plan COCO Annotator COCO Annotator is a web-based image annotation tool, crafted by Justin Brooks under the MIT license. Specifically designed to streamline the process of labeling images for object detection, localization, and keypoints detection models, this tool offers a range of features that cater to the diverse needs of machine learning practitioners and researchers.  COCO Annotator - Image Annotation Tool Key Features Image Annotation: Supports annotation of images for object detection, instance segmentation, keypoint detection, and captioning tasks. Export Formats: To facilitate large-scale object detection, the tool exports and stores annotations in the COCO format.  Automations: The tool makes annotating an image easier by incorporating semi-trained models. Additionally, it provides access to advanced selection tools, including the MaskRCNN, Magic Wand and DEXTR. Best For ML Research Teams: COCO Annotator is a good choice for ML researchers, preferable for image annotation for tasks like object detection and keypoints detection. Price Free Make Sense Make Sense AI is a user-friendly and open-source annotation tool, available under the GPLv3 license. Accessible through a web browser without the need for advanced installations, this tool simplifies the annotation process for various image types. Make Sense - Image Annotation Tool Key Features Open Sourced: Make Sense AI stands out as an open-source tool, freely available under the GPLv3 license, fostering collaboration and community engagement for its ongoing development. Accessibility: It ensures web-based accessibility, operating seamlessly in a web browser without complex installations, promoting ease of use across various devices. Export Formats: It facilitates exporting annotations in multiple formats (YOLO, VOC XML like Pascal VOC, VGG JSON, and CSV), ensuring compatibility with diverse machine learning algorithms and seamless integration into various workflows. Best For Small teams seeking an efficient solution to annotate an image. Price Free VGG Image Annotator VGG Image Annotator (VIA) is a versatile open-source tool crafted by the Visual Geometry Group (VGG) for the manual annotation of both image and video data. Released under the permissive BSD-2 clause license, VIA serves the needs of both academic and commercial users, offering a lightweight and accessible solution for annotation tasks. VGG Image Annotator - Image Annotation Tool Key Features Lightweight and User-Friendly: VIA is a lightweight, self-contained annotation tool, utilizing HTML, Javascript, and CSS without external libraries, enabling offline usage in modern web browsers without setup or installation. Offline Capability: The tool is designed to be used offline, providing a full application experience within a single HTML file of size less than 200 KB.  Multi-User Collaboration: Facilitates collaboration among multiple annotators with features such as project sharing, real-time annotation, and version control. Best For VGG Image Annotator (VIA) is ideal for individuals and small teams involved in projects for academic researchers. Price Free LabelMe LabelMe is an open-source web-based tool developed by the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) that allows users to label and annotate images for computer vision research. It provides a user-friendly interface for drawing bounding boxes, polygons, and semantic segmentation masks to label objects within images. LabelMe Image Annotation Tool Key Features Web-Based: Accessible through a web-based interface, allowing for annotation tasks to be performed in any modern web browser without requiring software installation. Customizable Interface: Provides a customizable annotation interface with options to adjust settings, colors, and layout preferences to suit specific project requirements. Best for Academic and research purposes Pricing Free Amazon SageMaker Ground Truth Amazon SageMaker Ground Truth is a fully managed data labeling service provided by Amazon Web Services (AWS). It offers a platform for efficiently labeling large datasets to train machine learning models. Ground Truth supports various annotation tasks, including image classification, object detection, semantic segmentation, and more. Amazon SageMaker Ground Truth - Image Annotation Tool Key Features Managed Service: Fully managed by AWS, eliminating the need for infrastructure setup and management. Human-in-the-Loop Labeling: Harnesses the power of human feedback across the ML lifecycle to improve the accuracy and relevancy of models. Scalability: Capable of handling large-scale annotation projects and accommodating growing datasets and annotation needs. Integration with Amazon SageMaker: Seamlessly integrates with Amazon SageMaker for model training and deployment, providing a streamlined end-to-end machine learning workflow. Best for Teams requiring large-scale data labeling. Pricing Varies based on labeling task and type of data. VOTT VOTT or Visual Object Tagging Tool is an open-source tool developed by Microsoft for annotating images and videos to create training datasets for computer vision models. VOTT provides an intuitive interface for drawing bounding boxes around objects of interest and labeling them with corresponding class names. VOTT Image Annotation Tool Key Features Versatile Annotation Tool: Supports a wide range of annotation types including bounding boxes, polygons, polylines, points, and segmentation masks for precise labeling. Video Annotation: Enables annotation of videos frame by frame, with support for object tracking and interpolation to streamline the annotation process. Multi-Platform Compatibility: Works across various operating systems such as Windows, Linux, and macOS, ensuring flexibility for users. Best for Teams requiring lightweight and customizable annotation tool for object detection. Pricing Free Image Annotation Tool: Key Takeaways There you have it! The 17 Best Image Annotation Tools for computer vision in 2024.  For further reading, you might also want to check out a few 2024 honorable mentions, both paid and free annotation tools: Supervisely - commercial data labeling platform praised for its quality control functionality and basic interpolation feature. Labelimg - Labelimg is an open source multi-modal data annotation tool now part of Label Studio. MarkUp - MarkUp image is a free web annotation tool to annotate an image or a PDF.