Software To Help You Turn Your Data Into AI
Forget fragmented workflows, annotation tools, and Notebooks for building AI applications. Encord Data Engine accelerates every step of taking your model into production.
Exploring the TACO Dataset [Data Analysis]
January 11, 2023
•5 min read
Introduction
Imagine for a moment you are building an autonomous robot that can collect litter on the ground. The robot needs to detect the litter to collect them and you – as a machine learning engineer – should build a litter detection system. This project, like most real-life computer vision problems, is challenging due to the variety of objects and backgrounds. In this technical tutorial, we will show you how Encord Active can help you in this process.
We will use Encord Active to analyze the dataset, the labels, and the model predictions. In the end, you will be know how to build better computer vision model with a data-centric approach.
TLDR;
- We set out to analyze the data and label quality of the TACO dataset and prepare it for model training. Whereafter we trained a Mask-RCNN model and analyzed the performance of our model.
- The dataset required significant preprocessing prior to training. It contains various outliers and the images are very large. Furthermore, when inspecting class distribution we found a high class imbalance.
- We found that most of the objects are very small. In addition, the annotation quality of the crowd-sourced dataset is a lot worse than the official dataset; therefore, annotations should be reviewed.
- Our object detection model perform well on larger objects which are distinct from the background and worse on small undefined objects.
Lastly, we inspected overall and class-based performance metrics. We learned that object area, frame object density, and object count have the highest impact on performance over others, and thus we should pay attention to those when retraining the model.
Okay! Lets go.
The Trash Annotations in Context (TACO) Dataset 🌮
For the tutorial we will use the open-source TACO dataset. The dataset repository contains a official dataset with 1500 images and 4784 annotations and an unofficial dataset with 3736 images and 8419 annotations. Both datasets contain various large and small litter objects on different backgrounds such as streets, parks, and beaches. The dataset contains 60 categories. of litter.
Example images from the official dataset:
About the dataset
- Research Paper: TACO: Trash Annotations in Context for Litter Detection
- Author: Pedro F Proença, Pedro Simões
- Dataset Size: Official: 1500 images, 4784 annotations & Unofficial: 3736 images, 8419 annotations.
- Categories: 60 litter categories
- License: CC BY 4.0
- Release: 17 March 2020
- Read more: Github & Webpage
Downloading the Dataset
So, let’s download the dataset from the official website by following the instructions. Currently, there are two datasets, namely, TACO-Official and TACO-Unofficial.
- The annotations of the official dataset are collected, annotated, and reviewed by the creators of the TACO project.
- The annotations of the unofficial dataset are provided by the community and they are not reviewed.
In this project, we will use the official dataset as a validation set and the unofficial dataset as a training set.
Installing Encord Active
To directly create a conda virtual environment including all packages, check the readme of the project.
Importing the TACO Dataset to Encord Active
You can import a COCO project to Encord Active with a single line command:
encord-active import project --coco -i ./images_folder -a ./annotations.json
We will run the above command twice (once for each dataset). The flow might take a while depending on the performance of your machine, so sit back and enjoy some sweet tunes.
This command will create a local Encord Active project and pre-compute all your quality metrics.
Quality metrics are additional parameterizations added onto your data, labels, and models; they are ways of indexing your data, labels, and models in semantically interesting and relevant ways. In Encord Active, there are different pre-computed metrics that you can use in your projects. You can also write your own custom metrics.
Once the project is imported you will have a local Encord Active project and we can start to understand and analyse the dataset.
Understanding Data and Label Quality
First, let’s analyze the official dataset. To start the Encord Active application, we run the following command for the official dataset:
# from the projects root encord-active visualise OR # from anywhere encord-active visualise --target /path/to/project/root
Your browser will open a new window with Encord Active.
Data Quality
Now let’s analyze the dataset. First, we navigate to the Data Quality → Explorer tab and check the distribution of samples with respect to different image-level metrics.
Area: The Images in the dataset are fairly large, where the median image size is around 8 megapixels (roughly 4000x2000). So, they need to be processed properly before the training. Otherwise reading them from the disk will create a serious bottleneck in the data loader. In addition, we can see on the plot below that there are a few outliers of very large images and very small images. Those can be excluded from the dataset.
Aspect Ratio: Most images have an aspect ratio of around 0.8, which brings to mind that they are mostly taken with mobile phone cameras (in a vertical direction) which may create a potential bias when going to production.
Brightness: Most images are taken during the daytime, and there are a few images taken during the evening (when it is a bit darker). So, this will potentially affect the performance of the model when it is run in environments where it is darker.
Label Quality
Next, we will analyze the label quality. Go to the Label Quality → Explorer tab and analyze the object-level metrics here:
Class distribution: In total, there are 5038 annotations for 59 classes. It is clear that there is a high class imbalance with some classes having fewer than 10 annotations, and some more than 500.
Frame Object Density: In most of the images, objects only cover a small part of the image, which means annotation sizes are small compared to the image size. Thus it is important we choose a model accordingly (AKA avoid using models that have problems with small objects).
Object Annotation Quality: This metric gives a score for each object according to its closest neighbors in the embedding space. If the closest neighbors have a different label, the quality of the annotation will be low. We see that there are many samples with low object quality scores. This may mean several things: for example, different object classes may be semantically very close to each other, which makes them closer in the embedding space, or there might be annotation errors. These should be analyzed, cleaned, and fixed before model training.
Another interesting point is that when the average object annotation quality scores are compared between the unofficial and official datasets (mean score under Annotator statistics), there is a huge difference (0.195 vs. 0.336). Average scores show that the unofficial dataset has lower annotation quality, so a review of the annotations should be performed before the training.
Distribution of the object annotation quality for the official dataset
Distribution of the object annotation quality for the unofficial dataset
Object Area - Relative: Most of the objects are very small compared to the image. So, for this problem, images can be tiled or frameworks like SAHI can be used.
Takeaways and Insights
Based on the preliminary analysis we can draw following conclusions:
- The images are very large with few outliers, thus we should downsize the images prior to model training and remove the outliers.
- The aspect ratio of most images show that they are probably taken on phone cameras in a vertical perspective. This could potential bias our inference if the edge devices are different.
- A underrepresentation of dark images (night time) can have an effect on our litter detection system at night in a production environment.
- The dataset has a few overrepresented classes (cigarettes, plastic film, unlabeled litter) and many categories with below 20 annotations.
- The annotation quality is significantly worse in the unofficial dataset compared to the official dataset. Whether this is due to label errors or otherwise needs to be investigated.
Next let us start to prepare the data for training :)
Preparing Data for Model Training
First, let’s filter and export the data and annotations to create COCO annotations. Although the datasets already contain COCO annotations, we want to generate a new COCO file where attributes are compatible with Encord Active so when we import our predictions to Encord Active, images can be matched.
- Go to Actions → Filter & Export. Do not filter the data.
- Click Generate COCO file, when COCO file is generated, click Download filtered data to download COCO annotations.
- We perform this operation for both the unofficial and the official dataset.
In every batch, images should be loaded from the disk, however, as we concluded above, they are too large to be loaded efficiently. Therefore, to avoid creating any bottlenecks in our pipeline, we downscale them before starting the training. An example of this can be seen in the project folder utils/downscale_dataset.py. We have scaled all the images to 1024x1024. Loaded images will also be scaled internally by our model before feeding into the model.
Training a Model
For training, we chose to train Mask-RCNN architecture in the Torchvision package. Further, we used PyTorch to train it and Wandb to log the performance metrics.
Create a dataset class
We have implemented EncordMaskRCNNDataset class, which can be used with any COCO dataset generated from the Encord Active so it can be used not just for this project but any project generated from the Encord Active. It transforms the input and output into the format that MaskRCNN model wants:
class EncordMaskRCNNDataset(torchvision.datasets.CocoDetection):
def __init__(self, img_folder, ann_file, transforms=None):
super().__init__(img_folder, ann_file)
self._transforms = transforms
def __getitem__(self, idx):
img, target = super().__getitem__(idx)
img_metadata = self.coco.loadImgs(self.ids[idx])
image_id = self.ids[idx]
img_width, img_height = img.size
boxes, labels, area, iscrowd = [], [], [], []
for target_item in target:
boxes.append(
[
target_item["bbox"][0],
target_item["bbox"][1],
target_item["bbox"][0] + target_item["bbox"][2],
target_item["bbox"][1] + target_item["bbox"][3],
]
)
labels.append(target_item["category_id"])
area.append(target_item["bbox"][2] * target_item["bbox"][3])
iscrowd.append(target_item["iscrowd"])
segmentations = [obj["segmentation"] for obj in target]
masks = convert_coco_poly_to_mask(segmentations, img_height, img_width)
processed_target = {}
processed_target["boxes"] = torch.as_tensor(boxes, dtype=torch.float32)
processed_target["labels"] = torch.as_tensor(labels, dtype=torch.int64)
processed_target["masks"] = masks
processed_target["image_id"] = torch.tensor([image_id])
processed_target["area"] = torch.tensor(area)
processed_target["iscrowd"] = torch.as_tensor(iscrowd, dtype=torch.int64)
if self._transforms is not None:
img, processed_target = self._transforms(img, processed_target)
return img, processed_target, img_metadata
def __len__(self):
return len(self.ids)Create data loader
After defining the Torchvision dataset we create a dataset for the data loader:
dataset = EncordMaskRCNNDataset( img_folder=params.data.train_data_folder, ann_file=params.data.train_ann, transforms=get_transform(train=True), ) data_loader = torch.utils.data.DataLoader( dataset, batch_size=params.train.batch_size, shuffle=True, num_workers=params.train.num_worker, collate_fn=collate_fn, )
Create a training loop
The training loop is very straightforward. For each epoch, we train the model on the training set. Once the training is finished for that epoch, we measure the performance on training and validation sets and log the results. To make this training more elegant we have:
- Used a learning rate scheduler to decrease the learning rate when the performance is saturated.
- Used Adam optimizer to quickly converge.
- Only saved the best-performing model checkpoint.
- Used the TorchMetrics library to calculate mAP.
for epoch in range(params.train.max_epoch):
print(f"epoch: {epoch}")
train_one_epoch(model, device, data_loader, optimizer, log_freq=10)
if epoch % params.logging.performance_tracking_interval == 0:
train_map = evaluate(model, device, data_loader, train_map_metric)
val_map = evaluate(model, device, data_loader_validation, val_map_metric)
scheduler.step(val_map["map"])
if params.logging.wandb_enabled:
train_map_logs = {f"train/{k}": v.item() for k, v in train_map.items()}
val_map_logs = {f"val/{k}": v.item() for k, v in val_map.items()}
wandb.log(
{
"epoch": epoch + 1,
"lr": optimizer.param_groups[0]["lr"],
**train_map_logs,
**val_map_logs,
}
)
val_map_average = val_map["map"].cpu().item()
if val_map_average > best_map * (1 + 0.0001):
early_stop_counter = 0
best_map = val_map_average
print("overwriting the best model!")
if params.logging.wandb_enabled:
wandb.run.summary["best map"] = best_map
torch.save(
model.state_dict(),
os.path.join(wandb.run.dir, "best_maskrcnn.ckpt"),
)
else:
torch.save(model.state_dict(), "weights/best_maskrcnn.ckpt")
else:
early_stop_counter += 1
if early_stop_counter >= params.train.early_stopping_thresh:
print("Early stopping at: " + str(epoch))
break
print("Training finished")We have provided the supportive and utility functions and modules inside the project folder.
Before diving into the full model training, let’s check if the training pipeline works as expected. To check that, we run the training on a few samples and expect the model to overfit into those samples. This is a common trick to check if the training pipeline works from end to end as expected.
We have prepared a configuration file (config.ini) to set the training scripts. Once all the paths and training parameters are set correctly, we can start training:
(encord-maskrcnn) > python train.py
Model checkpoints are saved to the local wandb log folder so that we can use it later for inference.
Here is a sample log result for mAP and mAP@50 for both training and validation sets.
Parameters used for training
- Learning rate = 0.0001
- Batch size = 10
- Epoch = 30
- Num worker = 4
Importing Predictions to Encord Active
There are a few ways to import predictions into Encord Active, for a more detailed explanation please check here.
In this tutorial we will prepare a pickle file consisting of Encord Active Prediction objects, so they can be understood by the Encord Active. First, we need to locate the wandb folder to get the path of the model checkpoint. Every wandb experiment has a unique ID, which can be checked from the overview tab of the experiment on the wandb platform. Once you learned the wandb experiment ID, local checkpoint should be somewhere like this:
/path/to/project/wandb/run-[date_time]_[wandb_id]/files/best_maskrcnn.ckpt
Once you set up the inference section of the config.ini file, you can generate Encord Active predictions.
Here is the main loop to generate the pickle file:
model.eval()
with torch.no_grad():
for img, _, img_metadata in tqdm(
dataset_validation, desc="Generating Encord Predictions"
):
prediction = model([img.to(device)])
scores_filter = prediction[0]["scores"] > confidence_threshold
masks = prediction[0]["masks"][scores_filter].detach().cpu().numpy()
labels = prediction[0]["labels"][scores_filter].detach().cpu().numpy()
scores = prediction[0]["scores"][scores_filter].detach().cpu().numpy()
for ma, la, sc in zip(masks, labels, scores):
contours, hierarchy = cv2.findContours(
(ma[0] > 0.5).astype(np.uint8), cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE
)
for contour in contours:
contour = contour.reshape(contour.shape[0], 2) / np.array(
[[ma.shape[2], ma.shape[1]]]
)
prediction = Prediction(
data_hash=img_metadata[0]["data_hash"],
class_id=encord_ontology["objects"][la.item() - 1][
"featureNodeHash"
],
confidence=sc.item(),
format=Format.POLYGON,
data=contour,
)
predictions_to_store.append(prediction)
with open(
os.path.join(validation_data_folder, f"predictions_{wandb_id}.pkl"), "wb"
) as f:
pickle.dump(predictions_to_store, f)Once we have the pickle file, we run the following command to import predictions:
encord-active import predictions /path/to/predictions.pkl
Lastly, we refresh the Encord Active browser.
Understanding Model Quality
With a freshly trained model, we’re ready to gather some insights. Encord Active automatically matches your ground-truth labels and predictions and presents valuable information regarding the performance of your model. For example:
- mAP performance for different IoU thresholds
- Precision-Recall curve for each class and class-specific AP/AR results
- Which of the 25+ metrics have the highest impact on the model performance
- True positive and false positive predictions as well as false negative ground-truth objects
First we evaluate the model on a high-level.
Model Performance
As explained previously we used the unofficial dataset for the training and official dataset for validation.
For the entire dataset we achieved a mAP @ 0.5 IOU at 0.041.
The mAP is very low. This means that the model is having difficulty detecting litters. We have also trained a model where both training and validation sets are from the official dataset, and the results were around 0.11, which is comparable with the reported results in the literature. This result also shows how difficult our task is.
Important Quality Metrics
First we want to investigate what quality metrics impact model performance. High importance for a metric implies that a change in that quantity would strongly affect model performance:
Hmm... object area, frame object density, and object count are most important metrics for the model performance. We should investigate that.
Best performing objects
Not surprisingly, among the best performing object are larger and distinct object such as cans and bottles.
Worst performing objects
Worst performing objects are the ones which do not have a clear definition: other plastic, other cartoon, unlabeled litter.
Other model insights
Next we, go dive into the model performance by metric and examine performance with respect to different metrics. Here are some insights we got from there:
- False negative rate and object count (number of labels pr. image) are directly proportional.
- The model tends to miss ground-truth objects when the sharpness value of the image is low (Image is more blurry).
- The smaller an object is relative to the image the worse performance. This is especially true for objects whose area are less than 0.01% of the image.
True positives
Let’s examine a few true positive:
The model segments some objects very well. In this tab, we can rank all true positives according to a metric that we want. For example, if we want to visualize correctly segmented objects in small images, we can select the area metric on the top bar.
False positives
Let’s examine a few false positives:
We see some interesting patterns here. The model’s segmentation for the objects are quite good, however, they are regarded as false positives. This may mean two things: 1) the model is confused about the object class or 2) there is a missing annotation. After inspecting the ground truth labels, we found that there are some classes which are confused with each other (e.g., garbage bag vs. single use carrier bag, glass bottle vs. other plastic bottle).
Examples from picture below
- The glass bottle is detected as other plastic bottle.
- The paper cup has no ground truth annotation:
- The single use carrier bag is perfectly detected as a garbage bag.
If we detect frequently confused objects, we can provide more data for these object classes or we can add specific post-processing steps to our inference pipeline, which will eventually increase our overall performance. These points are all very valuable to know when building our litter detection system since they show us areas we need improvement in our computer vision pipeline.
Conclusion
With the help of the Encord Active, we now have a better understanding of what we have in our hands and we:
- Obtained useful information on images such as their area, aspect ratio, brightness levels, which might be useful in building the training pipeline.
- Inspected class distribution and learned that there is a high class imbalance.
- Found that most of the objects are very small. In addition, the annotation quality of the crowd-sourced dataset is a lot worse than the official dataset; therefore, annotations should be reviewed.
- Inspected overall and class-based performance metrics. We learned that object area, frame object density, and object count have the highest impact on performance over others, and thus we should pay attention to those when retraining the model.
- Visualized true positives and false positive samples and figured out that there is a class mismatch problem. If this is solved, it will significantly reduce the false negatives.
Further Steps
With the insights we got from Encord Active, we now know what should be prioritized. In the second part of this tutorial, we will utilize the information we obtained here to improve our baseline result with a new and improved dataset.
Want to test your own models?
"I want to get started right away" - You can find Encord Active on Github here.
"Can you show me an example first?" - Check out this Colab Notebook.
"I am new, and want a step-by-step guide" - Try out the getting started tutorial.
If you want to support the project you can help us out by giving a Star on GitHub :)
Want to stay updated?
- Follow us on Twitter and Linkedin for more content on computer vision, training data, and active learning.
- Join the Slack community to chat and connect.
Written by Görkem Polat
Görkem is a Machine Learning Engineer at Encord. He holds a PhD from the Middle East Technical University, focused on solving computer vision problems in gastroenterology. He also holds a Master's Degree in Electronics Engineering, during which he worked on the classification of lung nodul... see more
View more posts
Build better ML models with Encord
Get started today
Discuss this blog on Slack
Join the Encord Developers community to discuss the latest in computer vision, machine learning, and data-centric AI
Join the communityRelated Blogs
How To Use Encord’s Bitmask Brush Tool
In machine learning, precise image annotation is crucial for training accurate and reliable models. Encord's Bitmask brush tool revolutionizes the annotation process by allowing interactive and fine-grained selection of regions of interest within images. Designed to cater to the needs of machine learning practitioners, this comprehensive guide will walk you through the ins and outs of utilizing Encord's Bitmask brush tool, empowering you to create precise and highly accurate annotations within the Encord platform. What is the bit mask brush? A bit mask brush allows you to interactively define regions or areas of interest within an image by "brushing" over them. As you paint or brush over the image, the bit mask brush assigns specific ‘bits’ or values to the corresponding pixels or regions you select. These bits represent the labels or categories associated with the selected areas. Accessing brush tool: Click on 🖌️ or press ‘f’ For example, if you are labeling outlines of blood vessels in an image, you can use a bit of mask brush to brush over the pixels corresponding to the vessel’s boundaries. The bit mask brush would assign a specific value or bit pattern to those pixels, indicating that they belong to the vessel class or category. Similarly, if you are labeling topologically separate regions belonging to the same frame classification, you can use a bitmask brush to assign different bit patterns or values to the regions you select. This allows you to differentiate between regions or segments within the same frame category. Using the Bitmask Brush The Bitmask brush is a powerful tool for creating annotations or labels by selecting specific regions within an image, providing flexibility and control over the labeling process. Let’s explore its key functionalities: Selection and Size Adjustment When the Bitmask annotation type is selected, the brush tool is automatically chosen by default. You can access it by clicking the brush icon or pressing the 'f' key, and you are able to adjust the brush size using a convenient slider. This enables you to tailor the brush size to the level of detail needed for your annotations. Annotation Creation Once you have adjusted the brush size, you can begin annotating your image by selecting the desired areas. As you brush over the regions, the Bitmask brush assigns specific bit patterns or values to the corresponding pixels, indicating their association with the selected labels or categories. Apply Label Once your annotation is complete, you can apply the label by clicking the "Apply label" button or pressing the Enter key, finalizing the annotation and incorporating it into the labeling or annotation process. 💡To use the bitmap masks, the ontology should contain the Bitmask annotation type. Eraser The Eraser tool provides the ability to erase parts or the entirety of your bitmask selection. This can be useful if you need to refine or modify your annotations before applying the final label. You can access the Eraser tool by clicking the eraser icon or pressing the 'h' key on your keyboard while the popup window is open. Accessing eraser tool: Click on eraser icon or press ‘h’ Threshold Brush The Threshold brush, specific to DICOM images, offers additional functionality by enabling you to set an intensity value threshold for your labels. The preview toggle allows you to visualize which parts of the image correspond to your set threshold, helping you determine the areas that will be labeled when covered by the Threshold brush. To access the Threshold brush, click the corresponding icon or press the 'g' key while the popup window is open. Adjust the brush size and the range of intensity values using the sliders in the popup. Accessing threshold tool: Click on the corresponding icon or press ‘g’ With the Encord Bitmask SDK The Encord Bitmask SDK empowers you to effortlessly generate, modify, and analyze annotations within the Encord platform, leveraging the vast capabilities of Python's comprehensive libraries and tools to their fullest extent. Find more details in the bitmask documentation. To conclude, Encord’s Bitmask brush tool, equipped with its diverse range of features, offers an intuitive and flexible solution for creating annotations within the Encord platform. Harnessing the power of the Bitmask brush and the Encord Bitmask SDK, you can elevate your annotation workflow to achieve precise and reliable results. Recommended Articles Medical Image Segmentation: A Complete Guide 6 Best Open Source Annotation Tools for Medical Imaging Guide to Experiments for Medical Imaging in Machine Learning 7 Ways to Improve Medical Imaging Dataset Future for Computer Vision in Healthcare
June 23
Search Anything Model: Combining Vision and Natural Language in Search
In the current AI boom, one thing is certain: data is king. Data is at the heart of the production and development of new models; and yet, the processing and structuring required to get data to a form that is consumable by modern AI are often overlooked. One of the most primordial elements of intelligence that can be leveraged to facilitate this is search. Search is crucial to understanding data: the more ways to search and group data, the more insights you can extract. The greater the insights, the more structured the data becomes. Historically, search capabilities have been limited to uni-modal approaches: models used for images or videos in vision use cases have been distinct from those used for textual data in natural language processing. With GPT-4’s ability to process both images and text, we are only now starting to see the potential impacts of performant multi-modal models that span various forms of data. Embracing the future of multi-modal data, we propose the Search Anything Model. The unified framework combines natural language, visual property, similarity, and metadata search together in a single package. Leveraging computer vision processing, multi-modal embeddings, LLMs, and traditional search characteristics, Search Anything allows for multiple forms of structured data querying using natural language. If you want to find all bright images with multiple cats that look similar to a particular reference image, Search Anything will match over multiple index types to retrieve data of the requisite form and conditions. What is Natural Language Search? Natural Language Search (NLS) uses human-like language to query and retrieve information from databases, datasets, or documents. Unlike traditional keyword-based searches, NLS algorithms employ Natural Language Processing (NLP) techniques to understand the context, semantics, and intent behind user queries. By interpreting the query’s meaning, NLS systems provide more accurate and relevant search results, mimicking how humans communicate. The computer vision domain requires a similar general understanding of data content without requiring metadata for visuals. 💡Encord is a data-centric computer vision company. With Encord Active, you can use the Search Anything Model to explore, curate, and debug your datasets. What Can You Use the Search Anything Model for? Let’s dive into some examples of computer vision uses for the Search Anything Model. Data Exploration Search Anything simplifies data exploration by allowing users to ask questions in plain language and receive valuable insights. Instead of manually formulating complex queries and algorithms that may require pre-existing metadata, you can pose questions such as: “Which images are blurry?” Or “How is my model performing on images with multiple labels?” Search Anything interprets these queries to provide visualizations or summaries of the data quickly and effectively to gain valuable insights. Data Curation Search Anything streamlines data curation, making the process highly efficient and user-friendly. Filter, sort, or aggregate data using only natural language commands For example, you can request the following: “Remove all the very bright images from my dataset” Or “Add an ‘unannotated’ tag to all the data that has not been annotated yet.” Search Anything processes these commands, automatically performs the requested actions, and presents the curated data all without complex coding or SQL queries. Using Encord Active to filter out bright images in the COCO dataset. Use the bulk tagging feature to tag all the data. Data Debugging Search Anything expedites the process of identifying and resolving data issues. To investigate anomalies to inconsistencies, ask questions or issue commands such as: “Are there any missing values for the image difficulty quality metric?” Or “Find records that are labeled ‘cat’ but don’t look like a typical cat.” Once again, Search Anything analyzes the data, detects discrepancies, and provides actionable insights to assist you in identifying and rectifying data problems efficiently. 💡Read to find out how to find and fix label errors with Encord Active. Cataloging Data for E-commerce Search Anything can also enhance the cataloging process for e-commerce platforms. By understanding product photos and descriptions, Search Anything enable users to search and categorize products efficiently, users can ask: . “Locate the green and sparkly shoes.” Search Anything interprets this query, matches the desired criteria with the product images and descriptions, and displays the relevant products, facilitating improved product discovery and customer experience. How to Use Search Anything Model with Encord? At Encord, we are building an end-to-end visual data engine for computer vision. Our latest release, Encord Active, empowers users to interact with visual data only using natural language. Let’s dive into a few use cases: Use Case 1: Data Exploration User Query: “red dress,” “denim jeans,” and “black shirts” Encord Active identifies the images in the dataset that most accurately corresponds to the query. Use Case 2: Data Curation User query: “Display the very bright images” Encord Active displays filtered results from the dataset based on the specified criterion. Read to find out how to choose the right data for your computer vision project. Use Case 3: Data Debugging User Query: “Find all the non-singular images?” Encord Active detects any duplicated images in the dataset, and displays images that are not unique within the dataset. Can I Use My Own Model? Yes, Encord Active allows you to leverage your models. Through fine-tuning or integrating custom embedding models, you can tailor the search capabilities to your specific needs, ensuring optimal performance and relevance. 💡At Encord, we are actively researching how to fine-tune LLMs for the purpose of searching Encord Active projects efficiently. Get in touch if you would like to get involved. Conclusion Natural Language Search is revolutionizing the way we interact with data, enabling intuitive and efficient exploration, curation, and debugging. By harnessing the power of NLP and computer vision models, our Search Anything Model allows you to pose queries, issue commands, and obtain actionable insights using human-like language. Whether you are an ML engineer, a data scientist, or an e-commerce professional, incorporating NLS into your workflow can significantly enhance productivity and unlock the full potential of your data.
June 20
How to Automate Data Labeling [Examples + Tutorial]
If you feed an AI model with junk, it’s bound to return the favor. The quality of the data being consumed by an AI algorithm has a direct correlation with its success when it comes to generalizing to new instances; this is the reason data professionals spend 80% of their time during model development, ensuring the data is appropriately prepared, and is representative of the real world. Data labeling is an essential task in supervised learning, as it enables AI algorithms to create accurate input-to-output mappings and build a comprehensive understanding of their environment. Data labeling can consume up to 80% of data preparation time, and at least 25% of an entire ML project is spent labeling. Therefore, efficient data labeling strategies are critical for improving the speed and quality of machine learning model development. 💡Read the blog to learn how to automate your data labeling process. Manual data labeling can be a challenging and error-prone process, as it relies on human judgment and subjective interpretation. Labelers may have different levels of expertise, leading to consistency in the labeling process and reduced accuracy. Moreover, manual data labeling can be time-consuming and expensive, especially for large datasets. This can hinder the scalability and efficiency of AI model development. Integrating automated data labeling into your machine learning projects can be an effective strategy for mitigating the challenges of manual data labeling. By leveraging AI technology to perform data labeling tasks, businesses can reduce the risk of human error, increase the speed and efficiency of model development, and minimize costs associated with manual labeling. Additionally, automated data labeling can help improve the accuracy and consistency of labeled data, resulting in more reliable and robust AI models. Let's take a closer look at automated data labeling, including its workings, advantages, and how Encord can assist you in automating your data labeling process. Using Annotation Tools for Automated Data Labeling Automated data labeling is using software tools and algorithms to automatically annotate or tag data with labels or tags that help identify and classify the data. This process is used in machine learning and data science to create training datasets for machine learning models. “Automated data annotation is a way to harness the power of AI-assisted tools and software to accelerate and improve the quality of creating and applying labels to images and videos for computer vision models.” – Frederik H. The Full Guide to Automated Data Annotation. Annotation tools can be used for automated data labeling by providing a user interface for creating and managing annotations or labels for a dataset. These tools can help to automate the process of labeling data by providing features such as: Auto-labeling: Annotation tools can use pre-built machine learning models or algorithms to generate labels for data automatically. Active learning: Annotation tools can use machine learning algorithms to suggest labels for data based on patterns and correlations in the existing labeled data. Human-in-the-loop: Annotation tools can provide a user interface for human annotators to review and correct the labels generated by the automation process. Quality control: Annotation tools can help to ensure the quality of the labels generated by the automation process by providing tools for validation and verification. Data management: Annotation tools can provide tools for managing and organizing large datasets, including tools for filtering, searching, and exporting data. Organizations can reduce the time and cost required to create high-quality training datasets for machine learning models by using annotation tools for automated data labeling. However, it is important to ensure that the tools used are appropriate for the specific task and that the labeled data is carefully validated and verified to ensure its quality. AI Annotation Tools 💡Check out our curated list of the 9 Best Image Annotation Tools for Computer Vision to discover what other options are on the market. Encord Annotate Encord Annotate is an automated annotation platform that performs AI-assisted image annotation, video annotation, and dataset management; part of the Encord product, alongside Encord Active. The key features of Encord Annotate include: Support for all annotation types such as bounding boxes, polygons, polylines, image segmentation, and more. It incorporates auto-annotation tools such as Meta’s Segment Anything Model and other AI-assisted labeling techniques. It has integrated MLOps workflow for computer vision and machine learning teams Use-case-centric annotations — from native DICOM & NIfTI annotations for medical imaging to SAR-specific features for geospatial data. Easy collaboration, annotator management, and QA workflows — to track annotator performance and increase label quality. Robust security functionality — label audit trails, encryption, FDA, CE Compliance, and HIPAA compliance. Benefits of Automated Data Labeling with AI Annotation Tools The most straightforward way to label data is to implement it manually, where a human user is presented with raw unlabeled data and applies a set of rules to label it. However, this approach has certain drawbacks such as being time-consuming and costly and having a higher probability of natural human error. An alternative approach is to use AI annotation tools to automate the labeling process, which can help address the issues associated with manual labeling by: Increasing accuracy and efficiency: Speed is just as important as being accurate. Yes, an automatic AI annotation tool can process large amounts of images much faster than a human can, but what makes it so effective is its ability to remain accurate, which ensures labels are precise and reliable. Improving productivity and workflow: It’s normal for humans to make mistakes – especially when they are performing the same task for 8 or more hours straight. When you use an AI-assisted labeling tool, the workload is significantly reduced, which means annotating teams can put more focus on ensuring things are labeled correctly the first time around. Reduction in labeling costs and resources: Deciding to manually annotate data means paying someone or a group of people to carry out the task; this means each hour that goes by has a cost, which can quickly become extremely high. An AI-assisted labeling tool may take off some of that load by allowing a human annotation team can manually label a percentage of the data and then have an AI tool do the rest. How to Automate Data Labeling with Encord A step-by-step guide to automating data labeling with Encord: Micro models Micro-models are models that are designed to be overtrained for a specific task or piece of data, making them effective in automating one aspect of data annotation workflow. They are not meant to be good at solving general problems and are typically used for a specific purpose. 💡Read the blog to find out more about micro-models. The main difference between a traditional model and a micro-model is not in their architecture or parameters but in their application domain, the data science practices used to create them, and their ultimate end-use. Step 1: Step 2: Auto-segmentation Auto-segmentation is a technique that involves using algorithms or annotation tools to automatically segment an image or video into different regions or objects of interest. This technique is used in various industries, including medical imaging, object detection, and scene segmentation. For example, in medical imaging, auto-segmentation can be used to identify and segment different anatomical structures in images, such as tumors, organs, and blood vessels. This can help medical professionals to make more accurate diagnoses and treatment plans Auto-segmentation can potentially speed up the image analysis process and reduce the likelihood of human error. However, it is important to note that the accuracy of auto-segmentation algorithms depends on the input data quality and the segmentation task's complexity. In some cases, manual review and correction may still be necessary to ensure the accuracy of the results. 💡Read the explainer blog on Segment Anything Model to understand how foundation models are used for auto-segmentation. Interpolation Interpolation is typically used to fill in missing values or smooth the noise in a dataset. It encompasses the process of estimating the value of a function at points that lie between known data points. Several methods can be used for interpolation in ML such as linear interpolation, polynomial interpolation, and spline interpolation. The choice of interpolation method will depend on the data's characteristics and the project's goals. Step 1: Step 2: Object Tracking Object tracking plays a vital role in various applications like security and surveillance, autonomous vehicles, video analysis, and many more. It’s a crucial component of computer vision that enables machines to track and follow objects in motion Using object tracking, you will be able to predict the position and other relevant information of moving objects in a video or image sequence. Step 1: Step 2: 💡Check out the Complete Guide to Object Tracking Tutorial to for more insight.. Conclusion Supervised machine learning algorithms depend on labeled data to learn how to generalize to unseen instances. The quality of data provided to the model has a significant impact on its final performance, hence it’s vital the data is accurately labeled and representative of the data available in a real-world scenario; this means AI teams often spend a large portion of their time preparing and labeling their data before it reaches the model training phase. Manually labeling data is slow, tedious, expensive, and prone to human error. One way to mitigate this issue is with automated data labeling and annotation solutions. Such tools can serve as a cost-effective way to accurately speed up the process, which in turn improves the team’s productivity and workflow. Ready to accelerate the automation of your data annotation and labeling? Sign-up for an Encord Free Trial: The Active Learning Platform for Computer Vision, used by the world’s leading computer vision teams. AI-assisted labeling, model training & diagnostics, find & fix dataset errors and biases, all in one collaborative active learning platform, to get to production AI faster. Try Encord for Free Today. Want to stay updated? Follow us on Twitter and LinkedIn for more content on computer vision, training data, and active learning. Join our Discord channel to chat and connect. Automated Data Labeling FAQs What are the benefits of automated data labeling? Automated data labeling helps to increase the accuracy and efficiency of the labeling process in contrast to when it’s performed by humans. It also reduces labeling costs and resources as you are not required to pay labelers to perform the tasks. How is automated data labeling different than manual labeling? Manual data labeling is the process of using individual annotators to assign labels to raw data. Opposingly, automated labeling is the same thing but the responsibility is passed on to machines instead of humans to speed up the process and reduce costs. What is AI data labeling? AI data labeling refers to a technique that leverages machine learning to provide one or more meaningful labels to raw data (e.g., images, videos, etc.). This is done with the intent of offering a machine learning model with context to learn input-output mappings from the data and make inferences on new, unseen data.
May 19
4 min