15 Interesting Github Repositories for Image Segmentation

Object tracking has become a fundamental part of the computer vision ecosystem. It powers various modern artificial intelligence applications and is behind several revolutionary technologies, such as self-driving cars, surveillance, and action recognition systems. Tracking algorithms use a combination of object detection and object tracking to detect and localize entities within a video frame. These algorithms range from basic machine learning to complex deep learning networks. Each of these has different implementations and use cases. This article will discuss the top 10 most popular video object-tracking algorithms. It will go over video object-tracking algorithms' back-end implementations, advantages, and disadvantages. We will also explore popular computer vision applications for object tracking. What is Video Object Tracking? Video object tracking refers to detecting an object within a video frame and tracking its position throughout the video. The concept of object tracking stems from object detection, a popular computer vision (CV) technique used for identifying and localizing different objects in images. While object detection works on still images (single frames), video object tracking applies this concept to every frame in the video. It analyzes each frame to identify the object in question and draw a bounding box around it. The object is effectively tracked throughout the video by performing this operation on all frames. However, complex machine learning and deep learning algorithms apply additional techniques such as region proposal and trajectory prediction for real-time object inference. Object tracking algorithms have revolutionized several industries. It has enabled businesses to implement analytics and automation in various domains and led to applications like: Autonomous Vehicles: Tracking surrounding elements like pedestrians, roads, curbs. Automated Surveillance: Tracking people or illegal objects like guns and knives. Sports Analytics: Tracking the ball or players to create match strategies. Augmented Reality Applications: Tracking all objects in the visual field to superimpose the virtual elements. Customer Analysis in Retail: Tracking retail store customers to understand movement patterns and optimize shelf placement. Over the years, object tracking algorithms have undergone various improvements in-terms of accuracy and performance. Let’s discuss these in detail. Single-stage Object Detectors Vs. Two-stage Object Detectors Object detection is a crucial part of tracking algorithms. Hence, it is vital to understand them in detail. There are two main categories of object detectors: Single-stage and two-stage. Both these methodologies have proven to provide exceptional results. However, each offers different benefits, with the former having a lower inference time and the latter having better accuracy. Single-stage detectors perform faster since they rely on a single network to produce annotations. These models skip intermediate feature extraction steps, such as region proposal. They use the raw input image to identify objects and generate bounding box coordinates. One example of a single-stage detector is You only look once (YOLO). YOLO can generate annotations with a single pass of the image. Single Stage Vs. Two-Stage Detection Two-stage detectors, such as Fast R-CNN, comprise two networks. The first is a region proposal network (RPN) that analyzes the image and extracts potential regions containing the desired objects. The second network is a CNN-based feature extractor that analyzes the proposed regions. The latter identifies the objects present and outputs their bounding box coordinates.  Two-stage object detectors are computationally expensive compared to their single-stage counterparts. However, they produce more accurate results. Object Tracking Approaches Object tracking algorithms work on two granularity levels. These include: Single Object Tracking (SOT) SOT is used to track the location of a single object throughout the video feed. These detection-free algorithms depend on the user to provide a bounding box around the target object on the first frame. The algorithm learns to track the position and movement of the object present within the box. It localizes the object's shape, posture, and trajectory in every subsequent frame. Single object tracking is useful when the focus must be kept on a particular entity. Some examples include tracking suspicious activity in surveillance footage or ball-tracking in sports analytics. Popular SOT algorithms include Particle Filters and Siamese Networks. However, one downside of traditional SOT algorithms is that they are unsuitable for context-aware applications where tracking multiple objects is necessary. Multiple Object Tracking (MOT) MOT works on the same concept as SOT. However, multi-object tracking identifies and tracks multiple objects throughout a video instead of a single object. MOT algorithms use extensive training datasets to understand moving objects. Once trained, they can identify and track multiple objects within each frame. Modern deep-learning MOT algorithms, like DeepSORT, can even detect new objects mid-video and create new tracks for them while keeping existing tracks intact. Multiple-object tracking is useful when various objects must be analyzed simultaneously. For example, in virtual reality (VR) applications, the algorithm must keep track of all objects in the frame to superimpose the virtual elements. However, these algorithms are computationally expensive and require lengthy training time. Phases of Object Tracking Process Visual object tracking is a challenging process comprising several phases. Target Initialization: The first step is to define all the objects of interest using labels and bounding boxes. The annotations, which include the names and locations of all the objects to be tracked, are specified in the first video frame. The algorithm then learns to identify these objects in all the subsequent images or video sequences. Appearance Modelling: An object may undergo visual transformation throughout the video due to varying lighting conditions, motion blur, image noise, or physical augmentations. This phase of the object-tracking process aims to capture these various transformations to improve the model’s robustness. It includes constructing object descriptions and mathematical models to identify objects with different appearances. Motion Estimation: Once the object features are defined, motion estimation predicts the object’s position based on the previous frame data. This is achieved by leveraging linear regression techniques or Particle Filters. Target Positioning: Motion estimation provides an estimate of the object's position. The next step is to pinpoint the exact coordinates within the predicted region. This is accomplished using a greedy search, i.e., checking every possibility or a maximum posterior estimation that looks at the most likely place using visual clues. Criteria for Selecting a Video Object Tracking Algorithm The two primary criteria to evaluate object tracking methods are accuracy and inference time. These help determine the best algorithm for particular use cases. Let’s discuss these criteria in detail. Accuracy Tracking algorithms output two main predictions: object identity (label) and location (bounding box coordinates). The accuracy of these models is determined by evaluating both these predictions and analyzing how well it was able to identify and localize the object. Metrics like Accuracy, Precision, Recall, and F1-score help evaluate the model's ability to classify the found object. While accuracy provides a generic picture, precision, and recall judge the model based on occurrences of false positives and negatives. Metrics like intersection-over-union (IoU) are used for localization accuracy. IoU calculates how much the predicted bounding box coincides with its ground truth value. A higher value means higher intersection and, hence, higher accuracy. Intersection Over Union (IoU) Inference Time The second judgment criterion is the speed of inference. Inference time determines how quickly the algorithm processes a video frame and predicts the object label and location. It is often measured in frames-per-second (FPS). This refers to the amount of frames the algorithm can process and output every second. A higher FPS value indicates faster inference. Challenges in Object Tracking Object tracking techniques carry various benefits for different industries. However, implementing a robust tracking algorithm is quite challenging. Some key challenges include: Object Variety: The real world comes with countless objects. Training a generic tracking algorithm would require an extensive dataset containing millions of objects. For this reason, object tracking models are generally domain-specific, with even the vastest models trained on only a few thousand objects. Varying Conditions: Besides the object variety, the training data must also cover objects in different conditions. A single object must be captured in different lighting conditions, seasons, times of day, and from different camera angles. Varying Image Quality: Images from different lenses produce varying information in terms of color production, saturation, etc. A robust model must incorporate these variations to cover all real-world scenarios. Computation Costs: Handling large image or video datasets requires considerable expertise and computational power. Developers need access to top-notch GPUs and data-handling tools, which can be expensive. Training deep-learning-based tracking algorithms can also increase operational costs if you use paid platforms that charge based on data units processed.  Scalability: Training general-purpose object tracking models requires extensive datasets. The growing data volumes introduce scalability challenges as developers require platforms that can handle increasingly large volumes of data and can increase computation power to train larger complex models. Top Algorithms for Video Object Tracking Here is a list of popular object tracking algorithms, ranging from simple mathematical models to complex deep learning architectures. Kalman Filter Kalman filters estimate an object’s position and predict its motion in subsequent frames. They maintain an internal representation of the object's state, including its position, velocity, and sometimes acceleration. The filters use information from the object’s previous state and a mathematical model analyzing the object’s motion to predict a future state. The model accounts for any uncertainty in the object's motion (noise). It incorporates all the discussed factors and estimates the object’s current state to create a future representation. Advantages It is a mathematical model that does not require any training. It is computationally efficient. Disadvantage Subpar performance and capabilities compared to modern deep learning algorithms. The model works on various assumptions, such as constant object acceleration. The algorithm does not perform well in random motion scenarios. KCF (Kernelized Correlation Filters) KCF is a mathematical model that understands object features and learns to distinguish them from their background. It starts with the user providing a bounding box around the object in the first frame. Once feature understanding is complete, it uses correlation filters based on the kernel trick to construct a high-dimensional relationship between the features and the true object. It uses the correlation features in subsequent frames to scan around the object's last known location. The area with the highest correlation is predicted to contain the object. Advantages Fast Computation. Low Memory Requirement. Competitive Results in general cases. Disadvantages Traditional KCF faces challenges in conditions such as varying object scales or objects touching frame boundaries. DeepSORT The Deep Simple Online Realtime Tracking (DeepSORT) algorithm extends the original SORT algorithm. The original SORT algorithm used Kalman filters to predict object motion and the Hungarian algorithm for frame-by-frame data association. However, this algorithm struggles with occlusions and varying camera angles and can lose object tracking in such complex scenarios. DeepSORT Architecture DeepSORT uses an additional convolutional neural network (CNN) as a feature extractor. These are called appearance features as they learn to determine the object identity (appearance) in different scenarios and allow the algorithm to distinguish between moving objects. DeepSORT combines the information from Filtering and CNN to create a deep association metric for accurate detection. Advantages DeepSort’s simple yet efficient implementation provides real-time performance. The model is modular. It can support any detection network of the user's choice, such as YOLO or SSD. It maintains its detection during occluded environments and can distinguish between different objects in complex scenarios. Disadvantages Offline training of a separate detection network can be challenging and requires an extensive dataset for high accuracy. FairMOT The fair multi-object tracking (FairMOT) algorithm uses a pre-trained model like faster R-CNN for detecting objects in the video sequence. It then uses a neural network to extract features from the detected object. FairMOT Architecture These features are used to track the object across other frames. The branches share the same underlying architecture and receive equal weightage during training. The FairMOT algorithm treats all classes fairly and provides a balanced performance between the two tasks: detection and tracking. Advantages Provides balanced performance between tracking and detection. Improved tracking accuracy due to the re-identification branch (feature extraction branch) Disadvantage Computationally expensive due to the two neural network branches being trained. MDNet The multi-domain network (MDNet) is popular for learning across different domains. It consists of two modules. The first is a CNN architecture shared amongst all the video sequences, i.e., it is domain-independent and learns from the entire dataset. This consists of CNN layers and a few flattened, fully connected layers. MDNet Architecture The second part comprises parallel fully connected (FC) layers, each processing domain-specific information. If the data captures information from 5 domains, the second portion will have 5 FC layers. Each of these layers is independently updated during back-propagation depending on the domain of the target image. Advantages Excellent performance across different domains. The domain-specific branches can be fine-tuned on the fly if significant domain shifts are detected. Disadvantages If data is imbalanced, the model will display uneven performance across the different domains. YOLOv8 (You Only Look Once) YOLOv8 is a single stage-detector that ranks among the most popular object tracking algorithms. The YOLO family of models is based on a CNN architecture that learns to predict object labels and positions with a single pass of the image. Yolov8 Tasks Catalog The model v8 follows a similar architecture to its predecessor and consists of various CNN and fully connected layers. It is an anchor-free algorithm, which directly predicts the object’s center rather than an offset from a predefined anchor. Moreover, the algorithm can be used for classification, segmentation, pose estimation, object detection, and tracking. YOLOv8 extends its detection capabilities by providing a range of trackers. Two popular options amongst these are Bot-SORT and ByteTrack. All the trackers are customizable, and users can fine-tune parameters like confidence threshold and tracking area. Advantages The model covers various use cases, including tracking and segmentation. High accuracy and performance. Easy Python Interface. Disadvantages Trouble detecting small objects. YOLOv8 provides various model sizes, each trading performance for accuracy. Here’s all you need to know about the YOLO family of model. Read more about YOLO models for Object Detection Explained [YOLOv8 Updated] Siamese Neural Networks (SNNs) Siamese-based tracking algorithms consist of two parallel branches of neural networks. One is a template branch, which contains the template image (including the object bounding box information) and the next frame where the object is to be found. This branch consists of CNNs and pooling layers and extracts features from both images, such as edges, texture, and shape. A fully convolutional siamese network for object tracking The other is the similarity branch that takes the features from the template and search image. It calculates the similarity between the two images using algorithms like contrastive loss. The output of this network is the likelihood of the object being present at different positions in the image. The Siamese network has had various advancements over the years. The modern architectures include attention mechanisms and RPNs for improved performance. Advantages Multiple advancements, including SiamFC, SiamRPN, etc. Disadvantages Training two parallel networks leads to long training times. GOTURN (Generic Object Tracking Using Regression Networks) Generic Object Tracking Using Regression Networks (GOTURN) is a deep learning based offline learning algorithm. The framework accepts two images, a previous frame and a current frame. The previous frame contains the object at its center, and the image is cropped to 2 times the bounding box size. The current frame is cropped in the same location, but the object is off-center as it has supposedly moved from its position. GOTURN High-level Architecture The internal structure of the model consists of convolutional layers taken from the CaffeNet architecture. Each of the two frames is passed through these layers, and the output is concatenated and processed through a series of fully connected layers. The objective of the network is to learn features from the previous frame and predict the bounding box in the current. Advantages Excellent performance, even on CPU. Disadvantages Troubled in scenarios where only some part of the object is visible. Object tracking is highly affected by imbalanced training data. Want to know more about creating a fair dataset? Read more about Balanced and Imbalanced Datasets in Machine Learning [Full Introduction] TLD (Tracking, Learning, and Detection) TLD is a framework designed for long-term tracking of an unknown object in a video sequence. The three components serve the following purpose: Tracker: Predicts the object location in the next frame using information in the current frame. This module uses techniques like mean-shift or correlation filtering. Detector: Scans the input frame-by-frame for potential objects using previously learned object appearances. Learning: Observes the tracker's and the detector's performance and identifies their errors. It further generates training samples to teach the detector to avoid mistakes in the future. Tracking-Learning-Detection Advantages Real-time performance. Disadvantages Sensitive to illumination changes. Can lose track of the object if it is completely occluded in any frame. Can fail if the object appearance changes mid-video. Median Flow Tracker The Median Flow Tracker predicts object movement in videos by analyzing feature points across frames. It estimates optical flow, filters out unreliable measurements, and uses the remaining data to update the object's bounding box. Tracking using Median Flow Internally, it tracks motion in both forward and backward directions and compares the two trajectories. Advantages Works well for predictable motion. Disadvantage Fails in scenarios of abrupt and random motion. Applications of Video Object Tracking Video Object Tracking has important use-cases in various industries. These use-cases automate laborious tasks and provide critical analytics. Let's discuss some key applications. Autonomous Vehicles Market leaders like Tesla, Waymo, and Baidu are constantly enhancing their AI infrastructure with state-of-the-art algorithms and hardware for improved tracking. Modern autonomous vehicles use different cameras and robust neural processing engines to track the objects surrounding them. Video object tracking plays a vital role in mapping the car's surroundings. This feature map helps the vehicle distinguish between elements such as trees, roads, pedestrians, etc. Autonomous Harvesting Robots Object tracking algorithms also benefit the agriculture industry by allowing autonomous detection and harvesting of ready crops. Agri-based companies like Four Growers use detection and tracking algorithms to identify harvestable tomatoes and provide yield forecasting. They use the Encord annotation tool and a team of professional annotators to label millions of objects simultaneously. Using AI-assisted tools has allowed them to cut the data processing time by half. Sports Analytics Sports analysts use computer vision algorithms to track player and ball movement to build strategies. Video tracking algorithms allow the analysts to understand player weaknesses and generate AI-based analytics. The tracking algorithms can also be used to fix player postures to improve performance and mitigate injury risks. Traffic Congestion & Emission Monitoring System Computer vision is used to track traffic activity on roads and airports. The data is also used to manage traffic density and ensure smooth flow. Companies like Automotus use object tracking models to monitor curb activity and reduce carbon emissions. Their solution automatically captures the time a car spends on the curb, detects any traffic violations, and analyzes driver behavior. Vascular Ultrasound Analysis Object detection has various use cases in the healthcare domain. One of the more prominent applications is Ultrasound analysis for diagnosing and managing vascular diseases like Popliteal Artery Aneurysms (PAAs). CV algorithms help medical practitioners in detecting anomalous entities in medical imaging. The automated detection allows for further AI analysis, such as classification, and allows the detection of minute irregularities that might otherwise be ignored. Professional Video Editing Professional tools like Adobe Premiere Pro use object tracking to aid professional content creators. It allows creators to apply advanced special effects on various elements and save time creating professional edits. Customer Analysis in Retail Stores Tracking algorithms are applied in retail stores via surveillance cameras. They are used to detect and track customer movement throughout the store premises. The tracking data helps the store owner understand hot spots where the customers spend the most time. It also gives insights into customer movement patterns that help optimize product placement on shelves. Want to know the latest computer vision use cases? Learn more about the ten most exciting applications of computer vision in 2024 Video Object Tracking: Key Takeaways The computer vision domain has come a long way, and tasks like classification, segmentation, and object tracking have seen significant improvements. ML researchers have developed various algorithms for video object tracking, each of which holds certain benefits over the other. In this article, we discussed some of the most popular architectures. Here are a few takeaways: Object Tracking vs. Object Detection: Video object tracking is an extension of object detection and applies the same principles to video sequences. Multiple Categories of Object Tracking: Object tracking comprises various sub-categories, such as single object tracking, multiple object tracking, single-stage detection, and two-stage detection. Object Tracking Metrics: Object tracking algorithms are primarily judged on their inference time (frames-per-second) and tracking accuracy. Popular Frameworks: Popular tracking frameworks include YOLOv8, DeepSORT, GOTURN, and MDNet. Applications: Object tracking is used across various domains, including healthcare, autonomous vehicles, customer analysis, and sports analytics.

Panoptic Segmentation Updates in Encord Over the past 6 months, we have updated and built new features within Encord with a strong focus on improving your panoptic segmentation workflows across data, labeling, and model evaluation. Here are some updates we’ll cover in this article: Bitmask lock. SAM + Bitmask lock + Brush for AI-assisted precision labeling. Fast and performant rendering of fully bitmask-segmented images and videos. Panoptic Quality model evaluation metrics. Bitmask Lock within Encord Annotate to Manage Segmentation Overlap Our Bitmask Lock feature introduces a way to prevent segmentation and masks from overlapping, providing pixel-perfect accuracy for your object segmentation tasks. By simply toggling the “Bitmask cannot be drawn over” button, you can prevent any part of a bitmask label from being included in another label. This feature is crucial for applications requiring precise object boundaries and pixel-perfect annotations, eliminating the risk of overlapping segmentations. Let’s see how to do this within Encord Annotate: Step 1: Create your first Bitmask Initiating your labeling process with the Bitmask is essential for creating precise object boundaries. If you are new to the Bitmask option, check out our quickstart video walkthrough on creating your first Bitmask using brush tools for labeling. Step 2: Set Bitmask Overlapping Behavior  Managing how bitmasks overlap is vital for ensuring accurate segmentation, especially when dealing with multiple objects that are close to each other or overlapping. After creating your first bitmask, adjust the overlapping behavior settings to dictate how subsequent bitmasks interact with existing ones. This feature is crucial for delineating separate objects without merging their labels—perfect for panoptic segmentation. This prevents any part of this bitmask label from being included in another label. This is invaluable for creating high-quality datasets for training panoptic segmentation models. Step 3: Lock Bitmasks When Labeling Multiple Instances Different images require different approaches. Beyond HSV, you can use intensity values for grayscale images (like DICOM) or RGB for color-specific labeling. This flexibility allows for tailored labeling strategies that match the unique attributes of your dataset. Experiment with the different settings (HSV, intensity, and RGB) to select the best approach for your specific labeling task. Adjust the criteria to capture the elements you need precisely. Step 4: Using the Eraser Tool Even with careful labeling, adjustments may be necessary. The eraser tool can remove unwanted parts of a bitmask label before finalizing it, providing an extra layer of precision. If you've applied a label inaccurately, use the eraser tool to correct any errors by removing unwanted areas of the bitmask. See our documentation to learn more. Bitmask-Segmented Images and Videos Got a Serious Performance Lift (At Least 5x) Encord's commitment to enhancing user experience and efficiency is evident in the significant performance improvements made to the Bitmask-segmented annotation within the Label Editor. Our Engineering team has achieved a performance lift of at least 5x by directly addressing user feedback and pinpointing critical bottlenecks. This improves how fast the editor loads for your panoptic segmentation labeling instances.  Here's a closer look at the differences between the "before" and "after" scenarios, highlighting the advancements: Before the Performance Improvements: Performance Lag on Zoom: Users experienced small delays when attempting to zoom in on images, with many instances (over 100) that impacted the precision and speed of their labeling process. Slow Response to Commands: Basic functionalities like deselecting tools or simply navigating through the label editor were met with sluggish responses. Operational Delays: Every action, from image loading to applying labels, was hindered by "a few milliseconds" of delay, which accumulated significant time overheads across projects. After the Performance Enhancements: Quicker Image Load Time: The initial step of image loading has seen a noticeable speed increase! This sets a good pace for the entire labeling task. Responsiveness: The entire label editor interface, from navigating between tasks to adjusting image views, is now remarkably more responsive. This change eradicates previous lag-related frustrations and allows for a smoother user experience. Improved Zoom Functionality: Zooming in and out has become significantly more fluid and precise. This improvement is precious for detailed labeling work, where accuracy is paramount. The positive changes directly result from the Engineering team's responsiveness to user feedback. Our users have renewed confidence in handling future projects with the Label Editor. We are dedicated to improving Encord based on actual user experiences. Use Segment Anything Model (SAM) and Bitmask Lock for High Annotation Precision Starting your annotation process can be time-consuming, especially for complex images. Our Segment Anything Model (SAM) integration offers a one-click solution to create initial annotations. SAM identifies and segments objects in your image, significantly speeding up the annotation process while ensuring high accuracy. Step 1: Select the SAM tool from the toolbar with the Bitmask Lock enabled.  Step 2: Click on the object you wish to segment in your image. SAM will automatically generate a precise bitmask for the object. Step 3: Use the bitmask brush to refine the edges for pixel-perfect segmentation if needed. See how to use the Segment Anything Model (SAM) within Encord in our documentation.   Validate Segmentation with Panoptic Quality Metrics You can easily evaluate your segmentation model’s panoptic mask quality with new metrics:  mSQ (mean Segmentation Quality) mRQ (mean Recognition Quality) mPQ (mean Panoptic Quality) The platform will calculate mSQ, mRQ, and mPQ for your predictions, labels, and dataset to clearly understand the segmentation performance and areas for improvement. Navigate to Active → Under the Model Evaluation tab, choose the panoptic model you want to evaluate. Under Display, toggle the Panoptic Quality Metrics (still in beta) option to see the model's mSQ, mRQ, and mPQ scores. Fast Rendering of Fully Bitmask-Segmented Images within Encord Active The performance improvement within the Label Editor also translates to how you view and load panoptic segmentation within Active.  Try it yourself: Key Takeaways: Panoptic Segmentation Updates in Encord Here’s a recap of the key features and improvements within Encord that can improve your Panoptic Segmentation workflows across data and models: Bitmask Lock: This feature prevents overlaps in segmentation. it guarantees the integrity of each label, enhancing the quality of the training data and, consequently, the accuracy of machine learning models. This feature is crucial for projects requiring meticulous detail and precision. SAM + Bitmask Lock + Brush: The Lock feature allows you to apply Bitmasks to various objects within an image, which reduces manual effort and significantly speeds up your annotation process. The integration of SAM within Encord's platform, using Lock to manage Bitmask overlaps, and the generic brush tool empower you to achieve precise, pixel-perfect labels with minimal effort. Fast and Performant Rendering of Fully Bitmask-segmented Images and Videos: We have made at least 5x improvements to how Encord quickly renders fully Bitmask-segmented images and videos across Annotate Label Editor and Active. Panoptic Quality Model Evaluation Metrics: The Panoptic Quality Metrics—comprising mean Segmentation Quality (mSQ), mean Recognition Quality (mRQ), and mean Panoptic Quality (mPQ)—provide a comprehensive framework for evaluating the effectiveness of segmentation models.

Let's take a moment to talk about one of the coolest things in AI right now: object segmentation. Simply put, object segmentation is about making sense of a picture by splitting it into its core parts and figuring out what's what. It’s kind of like solving a jigsaw puzzle, but instead of fitting pieces together, our AI is identifying and labeling things like cats, cars, trees, you name it! The practical implications of successful object segmentation are profound, spanning diverse industries and applications - from autonomous vehicles distinguishing between pedestrians and road infrastructure to medical imaging software that identifies and isolates tumors in a patient's scan to surveillance systems that track individuals or objects of interest. Each of these cases hinges on the capability to segment and identify objects within a broader context precisely; as a result, precise detection provides a granular understanding that fuels informed decisions and actions. Object segmentation is the secret sauce that makes this possible. Yet, as powerful as object segmentation is, it comes with its challenges. Conventionally, segmentation models require vast quantities of annotated data to perform at an optimal level. The task of labeling and annotating images can be labor-intensive, time-consuming, and often requires subject matter expertise, which can be a barrier to entry for many projects. At Encord, we utilize micro-models to speed up the segmentation process of computer vision projects. Users only label a few images from their dataset, and Encord micro-models learn the visual patterns of these labels by training on them. The trained micro-model is used to label the unlabeled images in the dataset, thereby automating the annotation process. As new foundational vision models such as DINO, CLIP, and Segment Anything Model emerge, the performance of these micro-models also increases. So we have decided to put two object segmentation models to the test to see how well they handle few-shot learning. We challenge these models to generate predictions for unlabeled images based on training a Mask-RCNN with 10 images and a recently proposed Personalized-SAM with 1 image, pushing the boundaries of what few-shot learning can achieve.  Segment Anything Model (SAM) Personalized-SAM uses the Segment Anything Model (SAM) architecture under the hood. Let’s dive into how SAM works. The Segment Anything Model (SAM) is a new AI model from Meta AI that can "cut out" any object, in any image, with a provided input prompt. SAM is a promptable segmentation system with zero-shot generalization to unfamiliar objects and images, without the need for additional training. So basically, SAM takes two inputs, an image and a prompt, and outputs a mask that is in relation to the prompt. The below figure shows the main blocks of the SAM architecture: Personalize Segment Anything Model with One Shot The image is passed through an image encoder which is a vision transformer model; this block is also the largest block of all SAM architecture. The image encoder outputs a 2D embedding (this being 2D is important here, as we will see later in Personalized-SAM). The prompt encoder processes the prompt, which can be points, boxes, masks, and texts. Finally, 2D image embedding and prompt embedding is fed to the mask decoder, which will output the final predicted masks. As there is a visual hierarchy of objects in images, SAM can output three different masks for the provided prompt. For a deep dive into the Segment Anything Model, read Meta AI's New Breakthrough: Segment Anything Model (SAM) Explained There are two main drawbacks of the SAM architecture: For each new image, a human is needed for the prompting. The segmented region is agnostic to the class; therefore, the class should be specified for each generated mask. Now, let’s move on to the Personalized-SAM to see how these issues are addressed. Personalized-SAM (Per-SAM) Personalized SAM (Per-SAM) is a training-free Personalization approach for SAM. The proposed approach works by first localizing the target concept in the image using a location prior, and then using a combination of target-guided attention, target-semantic prompting, and cascaded post-refinement to segment the target concept. Then, it uses a combination of target-guided attention, target-semantic prompting, and cascaded post-refinement to segment the target concept.  Personalize Segment Anything Model with One Shot Learning the target embedding First, the user provides an image and mask of the target concept (object). Then the image is processed through the image encoder block of the SAM architecture. Remember what the image encoder’s output was? Yes, it’s a 2D embedding! Now, one needs to find where the mask of the target object corresponds to in this 2D embedding space, as these embeddings will be the representation of our target concept. Since the spatial resolution is decreased in 2D embeddings, the mask should also be downscaled to that size. Then, the average of the vectors at locations that correspond to the downscaled mask should be obtained. Finally, we have a fixed embedding (or representation) for the target concept, which we will utilize in the next step. Getting a similarity map to generate prompts Now it is time for the inference on a new image. Per-SAM first processes it through the SAM image encoder to obtain a 2D embedding. The next step is crucial: finding the similarity between each location in the 2D embedding, and the target embedding that we obtained in the first step. The authors of Per-SAM use Cosine distance to calculate the similarity map. Once the similarity map is obtained, points for prompts can be generated for the prompt decoder. The maximum similarity location is determined as the positive point, and the minimum similarity location is determined as the negative point. So, now we have a prompt for the desired class for the new image. Can you guess what is next? We just need to apply the classic SAM process to get the mask. The image and prompts are given to the model and a segmentation mask is generated.  In the Per-SAM paper, the authors also use the similarity map as a semantic prompt. Moreover, they consecutively obtain new prompts (bounding box and segmented region) from the generated mask and refine the final predicted mask, which improves the overall result. Fine-tuning the Per-SAM Personalize Segment Anything Model with One Shot  In most of the images, objects have a conceptual hierarchy, which is already discussed in the SAM paper; therefore, SAM outputs 3 different masks when the prompt is ambiguous (especially for the point prompts). The authors of the Per-SAM propose another light layer, consisting of only 2 weights, on top of the SAM architecture to weigh the proposed masks. The entire SAM is frozen, and the generated masks are weighted with the learnable parameters, which results in rapid  (under 10 seconds) training. The reported results show that when fine-tuning is applied, the final performance result is significantly improved. Per-SAM vs. Mask R-CNN: Comparison Mask R-CNN (Mask Region-based Convolutional Neural Network) is a flexible and powerful framework for object instance segmentation, which is also one of the many deep learning models in Encord Apollo. Unlike SAM architecture, Mask R-CNN only accepts images as input and class annotations as output. Although it emerged a long time ago, it is still a very powerful model and is widely used in lots of applications and research papers as a baseline model. Therefore, comparing the Per-SAM against Mask R-CNN would be a sensible choice. Benchmark Dataset: DeepFashion-MultiModal  In this benchmark, we will use the DeepFashion-MutiModal dataset, which is a comprehensive collection of images created to facilitate a variety of tasks within the field of clothing and fashion analysis. It contains 44,096 high-resolution human images, including 12,701 full-body human images. All full-body images are manually annotated for 24 classes. The dataset also has key points, dense pose and textual description annotations. An image and its mask annotations The dataset is uploaded to Encord Active and here are more sample images and their annotations as seen in Encord Active platform: First, we have created a training set (10 images) and a test (inference) set (300 images) on the Encord platform. Encord platform provides two important features for this benchmark: We can visualize the images and labels easily. We can import predictions to the Encord Active platform which calculates the model performance. We have trained a MaskRCNN model on the training set using Encord Apollo. For the Per-SAM architecture, we have only used the first image in the training set. For the sake of simplicity, only ‘top’ type of class is used for this evaluation. The trained Mask-RCNN model and Per-SAM models are compared on the test images. Here is the benchmark table: Experiment results indicate that training-free Per-SAM outperforms the Mask-RCNN model, which shows the strong capability and clever approach of this new foundational model. Mask-RCNN and Per-SAM-1 shot performances are quite low which shows the difficulty of the dataset. When Per-SAM is trained on the image to weigh the masks, its performance is significantly improved. With fine tuning, Per-SAM learns the concept of the target object which results in a dramatic decrease in false positives and an increase in true positives. As a final step, we applied a post-processing step that removes the very small objects (if the object area to total area is less than 0.1%) in Per-SAM predictions. Which significantly reduced the false positives and resulted in a boost in the mAP score. When we examine the individual image results, we see that Per-SAM performs fairly well on most objects. Here are the true positives, false positives and false negatives: True Positives False Positives False Negatives Conclusion The experimental results show that new foundational models are very powerful regarding few-shot learning capabilities. Since their learning capacity is much more advanced than previous architectures and they are trained with millions of images (SAM is trained with over 11 million images with more than 1 billion masks), they can quickly learn new classes from a few images.  While the introduction of SAM was a milestone for computer vision, Per-SAM shows that there are many areas to explore on top of the SAM that will open new and more effective solutions. In Encord, we are constantly exploring new research and integrating them into the Encord platform to ease the annotation processes of our users.