In machine learning, training data plays a vital role in the accuracy and effectiveness of models. However, not all data is created equal, and the presence of outliers can significantly impact the performance of these models.  In this blog post, we will explore the concept of outlier detection and how it can be leveraged to improve training data with Encord Active. What is Outlier Detection? Outlier detection refers to identifying data points that deviate significantly from the normal distribution of a dataset. Outliers can arise due to various factors such as measurement errors, data corruption, or anomalies in the data. Detecting and handling outliers is crucial as they can distort statistical analysis and affect the performance of your ML models. {{Training_data_CTA}} In data analysis and machine learning, you can encounter two types of outliers:  Data outliers  Label outliers Data Outliers Data outliers refer to observations or instances in a dataset that significantly deviate from the expected or typical values. These outliers arise due to measurement errors, data corruption, or anomalies.  Data outliers can distort statistical analysis, affect the performance of machine learning models, and lead to inaccurate predictions. Detecting and addressing data outliers is crucial to ensure a high-quality dataset. Label Outliers Label outliers pertain to mislabeled or incorrectly assigned labels in a dataset. These outliers can occur due to human error during the labeling process or ambiguous instances that are challenging to classify accurately.  Label outliers can substantially impact the performance of supervised learning algorithms by introducing noise and misguiding the training process. Identifying and rectifying label outliers is essential for training models with accurate ground truth and improving their predictive capabilities. Both data outliers and label outliers require careful analysis and handling to ensure the quality and reliability of data for your machine learning tasks. You must employ robust outlier detection techniques and quality assurance procedures to identify and address these outliers for more accurate and dependable models. Outlier Detection in Encord Active Encord Active offers a robust solution to identify and label outliers for pre-defined metrics, custom metrics, and label classes using precomputed interquartile ranges  With this feature, you can easily spot data points that deviate significantly from the norm, enabling you to take appropriate actions and ensure data quality. By leveraging Interquartile ranges, Encord Active streamlines your outlier detection workflow, helping to debug your data. Setup To install Encord Active, follow these simple commands in your favorite Python environment: python3.9 -m venv ea-venv source ea-venv/bin/activate pip install encord-active {{gray_callout_start}} 💡 Check out the documentation for installation for more information. {{gray_callout_end}} Let's explore the steps in improving training data with outlier detection in Encord Active. Here we will be using the BDD dataset. {{gray_callout_start}} 💡 Check out the documentation for downloading the BDD dataset in Encord Active. {{gray_callout_end}} Data Outliers Finding the Data Outliers Encord Active provides an intuitive interface to locate outliers in your dataset. Navigate to the Data Quality > Summary tab and access the Quality Metrics, presented as expandable panes.  Click on a specific metric to reveal moderate to severe outliers, with the most severe ones displayed first, and use the slider to navigate through Tagging the Data Outliers Once you identify outliers of interest, Encord Active allows you to tag outliers individually or in bulk so you can easily manage and work with them for further analysis. Acting on the Data Outliers After creating the tagged image group, access it conveniently in the Actions tab at the bottom of the left sidebar with a range of actions at your disposal.  Select "Filter data frame on" and choose the "tags" option, to focus on the tagged outliers. You can then export the outliers, relabel them, augment the data, review them in detail, or even delete them from your dataset. Label Outliers Find Label Outliers To begin, navigate to the Label Quality > Summary tab in Encord Active. Here, you will find each Quality Metric presented as expandable panes, providing an overview of label quality. Click on a specific metric to gain deeper insights into moderate and severe label outliers. Like data outliers, the pane will prioritize presenting the severe outliers first, allowing you to focus on the most critical issues. Tag Label Outliers Once you have identified label outliers of interest, you can once again utilize individual and bulk tagging features to select and group the corresponding images. By tagging these outliers, you can conveniently organize and manipulate them for subsequent analysis and actions. Access the tagged image group at the bottom of the left sidebar in the Actions tab.  Act on Label Outliers Within the Actions tab, click "Filter data frame on" and select the "tags" option, allowing you to narrow down the data frame to focus solely on the tagged label outliers. From here, you can choose the desired actions, such as exporting the outliers, relabeling them, augmenting the dataset, reviewing them in detail, or even deleting them when necessary. How to Improve Training Data in Encord Active After reviewing the outlier detection procedure using Encord Active, let's examine the advantages it offers for enhancing training data.  Data Cleaning With its comprehensive set of tools for outlier detection, data visualization, and data quality assessment, Encord Active empowers users to identify problematic data points easily.  You can efficiently detect and address problematic data points early in your machine learning pipeline by leveraging these features. This proactive approach ensures that you can identify potential issues and mitigate them early on, leading to improved data quality and more reliable machine learning models. {{gray_callout_start}} 💡Encord allows you to filter data in 3 ways: Standard filter feature, embeddings plot, and natural language search. Click here to find out more. {{gray_callout_end}} Balancing Data Distribution The Actions tab in Encord Active offers two valuable options for refining your dataset: Filtering and creating a new dataset Balancing your existing dataset These options are accompanied by visualization capabilities, enabling you to make informed decisions about dataset quality without resorting to data augmentation or solely relying on the original dataset. By creating a small, balanced dataset through filtering, you can conduct thorough tests on your machine learning model. This approach helps you finalize your model in the pipeline, assess its performance, and determine if they require any further adjustments or augmentations. It also helps you evaluate different machine learning models. That enables you to make informed decisions about the most effective approach for your specific task. By leveraging the flexibility of the Actions tab in Encord Active and utilizing visualizations effectively, you can make data-driven choices regarding dataset balancing, augmentation, and model selection. This iterative process ensures that your training data is refined and optimized for the best possible model performance. {{gray_callout_start}} 💡To know more about the importance of data balancing, read the blog Introduction to Balanced and Imbalanced Datasets in Machine Learning {{gray_callout_end}} Continuous Iteration and Feedback Continuous iteration and refinement of training data based on model performance and feedback are crucial for achieving optimal results. Regularly evaluating the model's accuracy, identifying areas for improvement, and updating the training data accordingly are essential steps in this process.  Encord Active offers a range of tools to monitor model performance and assess the impact of data modifications, enabling informed decision-making. Model quality metrics provide a valuable means to evaluate data and labels using a trained model and imported predictions.  By leveraging these metrics, you can gain insights into the strengths and weaknesses of your dataset, enabling targeted improvements. Encord Active also helps with data versioning, allowing you to compare and assess the model's performance on each version. This iterative approach helps identify the best version of the dataset that yields optimal model performance. By leveraging the feedback loop between model evaluation, dataset refinement, and performance assessment in Encord Active, you can continuously optimize your training data to improve model quality. {{Training_data_CTA}} Conclusion Improving training data quality is crucial for boosting the accuracy and effectiveness of machine learning models. Outlier detection, as explored in this blog post, plays a vital role in identifying and managing data outliers and label outliers. Encord Active offers robust outlier detection features that enable users to easily spot significant deviations in data. By tagging and organizing outliers, Encord Active streamlines the workflow and ensures data quality through its intuitive interface and comprehensive toolset. However, the benefits of Encord Active extend beyond outlier detection. The platform empowers users to perform data cleaning, balance data distribution, and iterate on the dataset based on model performance and feedback. With data visualization, dataset filtering, and model quality metrics, users can make data-driven decisions and continually optimize their training data. By proactively addressing problematic data points, balancing data distribution, and iterating on the dataset, users can enhance the quality and reliability of their machine learning models. Encord Active serves as a powerful platform for these tasks, enabling users to refine and optimize their training data to achieve optimal model performance. Are you ready to improve your training data with Encord Active? Sign-up for a free trial of Encord: The Data Engine for AI Model Development, used by the world’s pioneering computer vision teams.  Follow us on Twitter and LinkedIn for more content on computer vision, training data, and active learning.

Data labeling operations are integral to the success of machine learning and computer vision projects. Data operation teams manage the entire end-to-end lifecycle of data labeling, including data sourcing, cleaning, and collaborating with ML teams to implement model training, quality assurance, and auditing workflows. The scalability of these teams is crucial. Behind the scenes, data operations teams ensure that artificial intelligence projects run smoothly.  As computer vision, machine learning, and deep learning projects scale and data volumes expand, it is critical that data ops teams grow, streamline, and adapt to meet the challenge of handling more labeling tasks.  In this article, we will cover 6 steps that data operations managers need to take to scale their teams and operational practices. What is Data Labeling for Machine Learning and Computer Vision? Data labeling or data annotation ⏤ the two terms that are often used synonymously, ⏤ is the act of applying labels and annotations to unlabeled data for the purpose of machine learning algorithms. Labels can be applied to various types of data, including images, video, text, and voice. For the purpose of this article, we will focus on data labeling for computer vision use cases, in which labels are applied to images and videos to create high-quality training datasets for AI models. Data labeling tasks could be as simple as applying a bounding box or polygon annotation with “cat” label or as complicated as microcellular labels applied to segmentations of tumors for a healthcare computer vision project.  Regardless of complexity, accuracy is essential in the labeling process to ensure high-quality training datasets and to optimize model performance.  Data labeling can be time-consuming and expensive. As such, companies must weigh the advantages and disadvantages of outsourcing or hiring in-house. While outsourcing is often more cost-effective, it comes with quality control concerns and data security risks. And, while in-house teams are expensive, they guarantee higher labeling quality and real-time insight into team members labeling tasks.  The quality of training data directly impacts the performance of machine learning algorithms.,, Ultimately, it comes down to the labeling quality, a responsibility entrusted to data labeling teams.  High-quality data requires a quality-centric data operations process with systems and management that can handle large volumes of labeling tasks for images or videos. {{gray_callout_start}} 💡Find out more with Encord’s guides on How to choose the best datasets for machine learning and How to choose the right data for computer vision projects. {{gray_callout_end}}  Challenges of Scaling Data Labeling Operations Data labeling is a time-consuming and resource-intensive function. Data ops team members have to account for and manage everything from sourcing data to data cleaning, building and maintaining a data pipeline, quality assurance, and training a model using training, validation, and test sets.  Even with an automated data annotation tool, there is a lot for data ops managers to oversee.  There are several challenges that data labeling teams face when scaling: Project resources: Scaling requires additional resources and funding. Determining the best allocation of both can be a challenge Hiring and training: Hiring and training new team members require time and resources to align with project requirements and data quality standards. This forces teams to consider the options of outsourcing or managing teams in-house?  Quality control: As the volume of data increases, maintaining How do we maintain high-quality labels becomes challenging.  Workflow and data security: As data labeling tasks increase, it can be challenging  to maintain data security, compliance, and audit trails. Annotation software: As image and video volumes increase, it can be challenging to manage projects. It is imperative to use the right tools, as teams can often benefit from the automation of data labeling tasks. Let’s look at how to solve these challenges.  6 Best Practices to Implement Scalable Data Labeling Operations Data operations teams are crucial for supporting data scientists and engineers. Here are 6 best practices for managing and implementing data labeling operations at scale. 1. Design a workflow-centric process Designing workflow-centric processes is crucial for any AI project. Data ops managers need to establish the data labeling project’s processes and workflows by creating standard operating procedures.  {{gray_callout_start}} 💡For more information, read Best Practice Guide for Computer Vision Data Operations Teams {{gray_callout_end}}  The support of senior leadership is vital to obtain the resources and budget to grow the data ops team, use the right tools, and employ a workforce for data labeling that can handle the volume needed. 2. Select an effective workforce for data labeling To select the appropriate workforce for data labeling operations, there are three options available: an in-house team, outsourced labeling services, or a crowd-sourced labeling team.  The choice depends on several factors:  Data volume Specialist knowledge Data security  Cost considerations Management In many cases, the benefits of using outsourced labeling service providers outweigh the associated risks and costs. In regulated sectors like healthcare, however, the use of in-house teams is often the only option given data security concerns and the highly specialized knowledge required.  Crowdsourcing through platforms like Amazon Mechanical Turk (MTurk) and SageMaker Ground Truth is another viable option for computer vision projects. Proper systems and processes, including workforce and workflow management and annotator training, are essential to the success of crowdsourcing or outsourcing.  3. Automate the data labeling process Similar to the staffing question, there are three options for automating data labeling: in-house tools, open-source, or commercial annotation solutions such as Encord. Open-source data labeling tools are suitable for projects with limited funding, such as academia or research, or for when a small team is building an MVP (minimum viable product) version of an AI model. These tools, however, often don’t meet the requirements for large-scale commercial projects. Developing an in-house tool can be a time-consuming and costly endeavor, taking 9 to 18 months and involving significant R&D expenses. In contrast, an off-the-shelf labeling platform can be quickly implemented. While pricing is higher than open-source (usually free for basic versions), it is cheaper than building an in-house data labeling tool. With an AI-assisted labeling and annotation platform, such as Encord, data ops teams can manage and scale the annotation workflows. The right tool also provides quality control mechanisms and training data-fixing solutions.  4. Leverage software principles for DataOps  Software development principles can be leveraged when scaling data labeling and training for a computer vision project.  Since data engineers, scientists, and analysts often engage in code-intensive tasks, integrating practices like continuous integration and delivery (CI/CD) and version control into data ops workflows is logical and advantageous.  {{try_encord_CTA_annotate_visual}} 5. Implement quality assurance (QA) iterative workflows  To ensure quality control and assurance at scale, it is crucial to establish a fast-moving and iterative process. One effective approach is to establish an active learning pipeline and dashboard. This allows data ops leaders to maintain tight control over quality at both a high-level and individual label level.  {{gray_callout_start}} 💡Here are our guides on 5 Ways to Improve The Quality of Data Labels and an Introduction to Quality Metrics in Computer Vision {{gray_callout_end}}  6. Ensure transparency and audibility in the data and labeling pipeline Label transparency and audibility are essential throughout the data pipeline.  A clear, user-logged, and timestamped audit trail is critical for projects in secure or regulated sectors like healthcare where  FDA compliance is required. With new AI laws likely to come into force worldwide in the next few years, a data labeling audit trail could also become mandatory for commercial AI models in non-regulated industries.   {{gray_callout_start}} 💡 Find out more with our Best Practice Guide for Computer Vision Data Operations Teams {{gray_callout_end}}   Scaling Data Labeling Operations: Key Takeaways  High-quality training datasets are essential for optimizing model performance. The function of data operations teams is to ensure the labeling quality and labeling workflow are smooth and frictionless.  Follow these 6 best practices to scale your data operations properly:  Design workflow-centric processes Select an effective workforce for data labeling  Automate the data labeling process Leverage software principles for DataOps  Implement QA iterative workflows  Ensure transparency and audibility in the data and labeling pipeline With an AI-powered annotation platform, data ops managers can oversee complex workflows, make annotation more efficient, and achieve labeling quality and productivity targets.   Are you ready to scale your data labeling operations and need a powerful AI-based software suite for computer vision projects?  Sign-up for a free trial of Encord: The Data Engine for AI Model Development, used by the world’s pioneering computer vision teams.  Follow us on Twitter and LinkedIn for more content on computer vision, training data, and active learning.

Meta recently released their new Visual Foundation Model (VFM) called the Segment Anything Model (SAM). An open-source VFM with powerful auto-segmentation workflows. Here’s our guide for how to use SAM to automate data labeling in Encord.  As data ops, machine learning (ML), and annotation teams know, labeling training data from scratch is time intensive and often requires expert labeling teams and review stages. Manual data labeling can quickly become expensive, especially for teams still developing best practices or annotating large amounts of data. Efficiently speeding up the data labeling process can be a challenge; this is where automation comes in. Incorporating automation into your workflow is one of the most effective ways to produce high-quality data fast. {{gray_callout_start}}If you want to learn about image segmentation, check out the full guide {{gray_callout_end}} Last week Meta released their new Visual Foundation Model (VFM) called the Segment Anything Model (SAM), an open-source VFM with incredible abilities to support auto-segmentation workflows. {{gray_callout_start}}If you want to learn about SAM, please read our SAM explainer.{{gray_callout_end}} Given this new release, Encord is excited to introduce the first SAM-powered auto-segmentation tool to help teams generate high-quality segmentation masks in seconds. Create quality masks with our SAM-powered auto-segmentation tool Whether you're new to labeling data for your model or have several models in production, our new SAM-powered auto-segmentation tool can help you save time and streamline your labeling process. To maximize the benefits of this tool, follow these steps: 1. Set up your image segmentation project by attaching your dataset and ontology.  2. Next, access our image auto-segmentation tool from the left-hand side toolbar. Steps 1 & 2 for using SAM with Encord 3. Prompt the SAM model with a key point or bounding box (Grid prompting coming soon, let us know if you want to help us test it out!) on or over the area you wish to segment. To speed up your workflow, you can draw multiple bounding boxes simultaneously, and our SAM-powered auto-segmentation tool will label them for you. Step 3 for using SAM with Encord 4. In a split second, our tool will calculate the image embeddings in the image encoder and generate a pixel-perfect segmentation mask for you using the prompt and prompt encoder. 5.  You can use our drawing/eraser tool to add to or subtract from the mask to adjust it.  If you accidentally create a mask over an area you don’t like, use the erase tool to erase it. Steps 4 & 5 for using SAM with Encord Integrate an AI-assisted Micro-model to make labeling even faster  While AI-assisted tools such as SAM-powered auto-segmentation can be great for teams just getting started with data labeling, teams who follow more advanced techniques like pre-labeling can take automation to the next level with micro-models. By using your model or a Micro-model, pre-labels can significantly increase labeling efficiency. As the model improves with each iteration, labeling costs decrease, allowing teams to focus their manual labeling efforts on edge cases or areas where the model may not perform as well. This results in faster, less expensive labeling with improved model performance. Check out our case study to learn more about our pre-labeling workflow, powered by AI-assisted labeling, and how one of our customers increased their labeling efficiency by 37x using AI-assisted Micro-models. Try our auto-segmentation tool on an image labeling project or start using model-assisted labeling today. {{SAM_CTA}} Key Takeaways Meta's Segment Anything Model (SAM) is a powerful and effective open-source Visual Foundation Model (VFM) that will make a positive difference to automated segmentation workflows for computer vision and ML projects. AI-assisted labeling can reduce labeling costs and improve with each iteration. Our SAM-powered auto-segmentation tool and AI-assisted labeling workflow are available to all customers. We're excited for our users to see how automation can significantly reduce costs and increase labeling efficiency. Ready to improve the performance and scale your data operations, labeling, and automated annotation?  Sign-up for an Encord Free Trial: The Active Learning Platform for Computer Vision, used by the world’s leading computer vision teams.  AI-assisted labeling, model training & diagnostics, find & fix dataset errors and biases, all in one collaborative active learning platform, to get to production AI faster. Try Encord for Free Today.  Want to stay updated? Follow us on Twitter and LinkedIn for more content on computer vision, training data, and active learning. Join the Slack community to chat and connect. Segment Anything Model (SAM) in Encord Frequently Asked Questions (FAQs) What is a segmentation mask? A segmentation mask assigns a label to each pixel in an image, identifying objects or regions of interest. This is done by creating a binary image where object pixels are marked with 1, and the rest with 0.  It's used in computer vision for object detection and image segmentation and for training machine learning models for image recognition. How does the SAM-powered auto-segmentation work? Combining SAM with Encord Annotate offers users an auto-segmentation tool with powerful ontologies, an interactive editor, and media support. SAM can segment objects or images without prior exposure using basic keypoint info and a bounding box. Encord annotate utilizes SAM to annotate various media types such as images, videos, satellite data, and DICOM data. If you want to better understand SAM’s inner workings and importance, please refer to the SAM explainer. Can I use SAM in Encord for bounding boxes? Encord’s auto-segmentation feature supports various types of annotations such as bounding boxes, polyline annotation, and keypoint annotation. Additionally, Encord utilizes SAM for annotating images, videos, and specializes data types including satellite (SAR), DICOM, NIfTI, CT, X-ray, PET, ultrasound and MRI files.  For more information on auto-segmentation for medical imaging computer vision models, please refer to the documentation. Can I Fine-tune SAM? The image encoder of SAM is a highly intricate architecture containing numerous parameters. To fine-tune the model, it is more practical to concentrate on the mask encoder instead, which is lightweight and, therefore simpler, quicker, and more efficient in terms of memory usage. Please read Encord’s tutorial on how to fine-tune Segment Anything. {{gray_callout_start}}You can find the Colab Notebook with all the code you need to fine-tune SAM here. Keep reading if you want a fully working solution out of the box!{{gray_callout_end}} Can I try SAM in Encord for free? Encord has integrated its powerful ontologies, interactive editor, and versatile data type support with SAM to enable segmentation of any type of data. SAM's auto-annotation capability can be utilized for this purpose. Encord offers a free trial that can be accessed by logging in to our platform or please contact our team to get yourself started 😀