Top 9 Tools for Generative AI Model Validation in Computer Vision

Today, machine learning (ML)-based forecasting has become crucial across various industries. It plays a pivotal role in automating business processes, delivering personalized user experiences, gaining a competitive advantage, and enabling efficient decision-making. A key component that drives decisions for ML systems is model inference. In this article, we will explain the concept of machine learning inference, its benefits, real-world applications, and the challenges that come with its implementation, especially in the context of responsible artificial intelligence practices. What is Model Inference in Machine Learning? Model inference in machine learning refers to the operationalization of a trained ML model, i.e., using an ML model to generate predictions on unseen real-world data in a production environment. The inference process includes processing incoming data and producing results based on the patterns and relationships learned during the machine learning training phase. The final output could be a classification label, a regression value, or a probability distribution over different classes. An inference-ready model is optimized for performance, efficiency, scalability, latency, and resource utilization. The model must be optimized to run efficiently on the chosen target platform to ensure that it can handle large volumes of incoming data and generate predictions promptly. This requires selecting appropriate hardware or cloud infrastructure for deployment, typically called an ML inference server. There are two common ways of performing inference:  Batch inference: Model predictions are generated on a chunk of observations after specific intervals. It is best-suited for low latency tasks, such as analyzing historical data.  Real-time inference: Predictions are generated instantaneously as soon as new data becomes available. It is best-suited for real-time decision-making in mission-critical applications. To illustrate model inference in machine learning, consider an animal image classification task, i.e., a trained convolutional neural network (CNN) used to classify animal images into various categories (e.g., cats, dogs, birds, and horses). When a new image is fed into the model, it extracts and learns relevant features, such as edges, textures, and shapes. The final layer of the model provides the probability scores for each category. The category with the highest probability is considered the model's prediction for that image, indicating whether it is a cat, dog, bird, or horse. Such a model can be valuable for various applications, including wildlife monitoring, pet identification, and content recommendation systems. Some other common examples of machine learning model inference include predicting whether an email is spam or not, identifying objects in images, or determining sentiment in customer reviews. Benefits of ML Model Inference Let’s discuss in detail how model inference in machine learning impacts different aspects of business. Real-Time Decision-Making Decisions create value – not data. Model inference facilitates real-time decision-making across several verticals, especially vital in critical applications such as autonomous vehicles, fraud detection, and healthcare. These scenarios demand immediate and accurate predictions to ensure safety, security, and timely action. A couple of examples of how ML model inference facilitates decision-making: Real-time model inference for weather forecasting based on sensor data enables geologists, meteorologists, and hydrologists to accurately predict environmental catastrophes like floods, storms, and earthquakes. In cybersecurity, ML models can accurately infer malicious activity, enabling network intrusion detection systems to actively respond to threats and block unauthorized access. Automation & Efficiency Model inference significantly reduces the need for manual intervention and streamlines operations across various domains. It allows businesses to take immediate actions based on real-time insights. For instance: In customer support, chatbots powered by ML model inference provide automated responses to user queries, resolving issues promptly and improving customer satisfaction. In enterprise environments, ML model inference powers automated anomaly detection systems to identify, rank, and group outliers based on large-scale metric monitoring. In supply chain management,real-timemodel inference helps optimize inventory levels, ensuring the right products are available at the right time, thus reducing costs and minimizing stockouts. Personalization Personalized Recommendation System Compared to Traditional Recommendation Model inference enables businesses to deliver personalized user experiences, catering to individual preferences and needs. For instance: ML-based recommendation systems, such as those used by streaming platforms, e-commerce websites, and social media platforms, analyze user behavior in real-time to offer tailored content and product recommendations. This personalization enhances user engagement and retention, leading to increased customer loyalty and higher conversion rates. Personalized marketing campaigns based on ML inference yield better targeting and improved customer response rates. Scalability & Cost-Efficiency End-to-end Scalable Machine Learning Pipeline By leveraging cloud infrastructure and hardware optimization, organizations can deploy ML applications cost-efficiently. Cloud-based model inference with GPU support allows organizations to scale with rapid data growth and changing user demands. Moreover, it eliminates the need for on-premises hardware maintenance, reducing capital expenditures and streamlining IT management. Cloud providers also offer specialized hardware-optimized inference services at a low cost. Furthermore, on-demand serverless inference enables organizations to automatically manage and scale workloads that have low or inconsistent traffic. With such flexibility, businesses can explore new opportunities and expand operations into previously untapped markets. Real-time insights and accurate predictions empower organizations to enter new territories with confidence, informed by data-driven decisions. Real-World Use Cases of Model Inference AI Technology Landscape Model inference in machine learning finds extensive application across various industries, driving transformative changes and yielding valuable insights. Below, we delve into each real-world use case, exploring how model inference brings about revolutionary advancements: Healthcare & Medical Diagnostics Model inference is revolutionizing medical diagnostics through medical image analysis and visualization. Trained deep learning models can accurately interpret medical images, such as X-rays, MRIs, and CT scans, to aid in disease diagnosis. By analyzing the intricate details in medical images, model inference assists radiologists and healthcare professionals in identifying abnormalities, enabling early disease detection and improving patient outcomes. Real-time monitoring of patient vital signs using sensor data from medical Internet of Things (IoT) devices and predictive models helps healthcare professionals make timely interventions and prevent critical events. Natural Language Processing (NLP) models process electronic health records and medical literature, supporting clinical decision-making and medical research. Natural Language Processing (NLP) Model inference plays a pivotal role in applications of natural language processing (NLP), such as chatbots and virtual assistants. NLP models, often based on deep learning architectures like recurrent neural networks (RNNs), long short-term memory networks (LSTMs), or transformers, enable chatbots and virtual assistants to understand and respond to user queries in real-time. By analyzing user input, NLP models can infer contextually relevant responses, simulating human-like interactions. This capability enhances user experience and facilitates efficient customer support, as chatbots can handle a wide range of inquiries and provide prompt responses 24/7. Autonomous Vehicles Model inference is the backbone of decision-making in computer vision tasks like autonomous vehicle driving and detection. Trained machine learning models process data from sensors like LiDAR, cameras, and radar in real-time to make informed decisions on navigation, collision avoidance, and route planning. In autonomous vehicles, model inference occurs rapidly, allowing vehicles to respond instantly to changes in their environment. This capability is critical for ensuring the safety of passengers and pedestrians, as the vehicle must continuously assess its surroundings and make split-second decisions to avoid potential hazards. Fraud Detection In the financial and e-commerce sectors, model inference is used extensively for fraud detection. Machine learning models trained on historical transaction data can quickly identify patterns indicative of fraudulent activities in real-time. By analyzing incoming transactions as they occur, model inference can promptly flag suspicious transactions for further investigation or block fraudulent attempts. Real-time fraud detection protects businesses and consumers alike, minimizing financial losses and safeguarding sensitive information. So, model interference can be used in horizontal and vertical B2B marketplaces, as well as in the B2C sector. Environmental Monitoring Model inference finds applications in environmental data analysis, enabling accurate and timely monitoring of environmental conditions. Models trained on historical environmental data, satellite imagery, and other relevant information can predict changes in air quality, weather patterns, or environmental parameters. By deploying these models for real-time inference, organizations can make data-driven decisions to address environmental challenges, such as air pollution, climate change, or natural disasters. The insights obtained from model inference aid policymakers, researchers, and environmentalists in developing effective strategies for conservation and sustainable resource management. Interested in learning more about ML-based environmental protection? Read how Encord has helped in Saving the Honey Bees with Computer Vision. Financial Services In the finance sector, ML model inference plays a crucial role in enhancing credit risk assessment. Trained machine learning models analyze vast amounts of historical financial data and loan applications to predict the creditworthiness of potential borrowers accurately. Real-time model inference allows financial institutions to swiftly evaluate credit risk and make informed lending decisions, streamlining loan approval processes and reducing the risk of default. Algorithmic trading models use real-time market data to make rapid trading decisions, capitalizing on market opportunities with dependencies. Moreover, model inference aids in determining optimal pricing strategies for financial products. By analyzing market trends, customer behavior, and competitor pricing, financial institutions can dynamically adjust their pricing to maximize profitability while remaining competitive. Customer Relationship Management In customer relationship management (CRM), model inference powers personalized recommendations to foster stronger customer engagement, increase customer loyalty, and drive recurring business. By analyzing customer behavior, preferences, and purchase history, recommendation systems based on model inference can suggest products, services, or content tailored to individual users. They contribute to cross-selling and upselling opportunities, as customers are more likely to make relevant purchases based on their interests. Moreover, customer churn prediction models help businesses identify customers at risk of leaving and implement targeted retention strategies. Sentiment analysis models analyze customer feedback to gauge satisfaction levels and identify areas for improvement. Predictive Maintenance in Manufacturing Model inference is a game-changer in predictive maintenance for the manufacturing industry. By analyzing real-time IoT sensor data from machinery and equipment, machine learning models can predict equipment failures before they occur. This capability allows manufacturers to schedule maintenance activities proactively, reducing downtime and preventing costly production interruptions. As a result, manufacturers can extend the lifespan of their equipment, improve productivity, and overall operational efficiency. Limitations of Machine Learning Model Inference  Model inference in machine learning brings numerous benefits, but it also presents various challenges that must be addressed for successful and responsible AI deployment. In this section, we delve into the key challenges and the strategies to overcome them: Infrastructure Cost & Resource Intensive Model inference can be resource-intensive, particularly for complex models and large datasets. Deploying models on different hardware components, such as CPUs, GPUs, TPUs, FPGAs, or custom AI chips, poses challenges in optimizing resource allocation and achieving cost-effectiveness. High computational requirements result in increased operational costs for organizations. To address these challenges, organizations must carefully assess their specific use case and the model's complexity. Choosing the right hardware and cloud-based solutions can optimize performance and reduce operational costs. Cloud services offer the flexibility to scale resources as needed, providing cost-efficiency and adaptability to changing workloads. Latency & Interoperability Real-time model inference demands low latency to provide immediate responses, especially for mission-critical applications like autonomous vehicles or healthcare emergencies. In addition, models should be designed to run on diverse environments, including end devices with limited computational resources. To address latency concerns, efficient machine learning algorithms and their optimization are crucial. Techniques such as quantization, model compression, and pruning can reduce the model's size and computational complexity without compromising model accuracy. Furthermore, using standardized model formats like ONNX (Open Neural Network Exchange) enables interoperability across different inference engines and hardware. Ethical Frameworks Model inference raises ethical implications, particularly when dealing with sensitive data or making critical decisions that impact individuals or society. Biased or discriminatory predictions can have serious consequences, leading to unequal treatment. To ensure fairness and unbiased predictions, organizations must establish ethical guidelines in the model development and deployment process. Promoting responsible and ethical AI practices involves fairness-aware training, continuous monitoring, and auditing of model behavior to identify and address biases. Model interpretability and transparency are essential to understanding how decisions are made, particularly in critical applications like healthcare and finance. Transparent Model Development Complex machine learning models can act as "black boxes," making it challenging to interpret their decisions. However, in critical domains like healthcare and finance, interpretability is vital for building trust and ensuring accountable decision-making. To address this challenge, organizations should document the model development process, including data sources, preprocessing steps, and model architecture. Additionally, adopting explainable AI techniques like LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHapley Additive exPlanations) can provide insights into how the model arrives at its decisions, making it easier to understand and interpret its behavior. Want to build transparent AI systems that comply with the latest regulations? Read our blog post: What the European AI Act Means for You, AI Developer. Robust Model Training & Testing During model training, overfitting is a common challenge, where the model performs well on the training data but poorly on unseen data. Overfitting can result in inaccurate predictions and reduced generalization. To address overfitting, techniques like regularization, early stopping, and dropout can be applied during model training. Data augmentation is another useful approach, i.e., introducing variations in the training data to improve the model's ability to generalize on unseen data. Want to learn more about handling ML datasets? Read our detailed guides on Introduction to Balanced and Imbalanced Datasets in Machine Learning and Training, Validation, Test Split for Machine Learning Datasets. Furthermore, the accuracy of model predictions heavily depends on the quality and representativeness of the training data. Addressing biased or incomplete data is crucial to prevent discriminatory predictions and ensure fairness. Additionally, models must be assessed for resilience against adversarial attacks and input variations. Adversarial attacks involve intentionally perturbing input data to mislead the model's predictions. Robust models should be able to withstand such attacks and maintain accuracy. Continuous Monitoring & Retraining Models may experience a decline in performance over time due to changing data distributions. Continuous monitoring of model performance is essential to detect degradation and trigger retraining when necessary. Continuous monitoring involves tracking model performance metrics and detecting instances of data drift. When data drift is identified, models can be retrained on the updated data to ensure their accuracy and relevance in dynamic environments. Security & Privacy Protection Model inference raises concerns about data and model security in real-world applications. Typically, four types of attacks can occur during inference: membership inference attacks, model extraction attacks, property inference attacks, and model inversion attacks. Hence, sensitive data processed by the model must be protected from unauthorized access and potential breaches. Ensuring data security involves implementing robust authentication and encryption mechanisms. Techniques like differential privacy and federated learning can enhance privacy protection in machine learning models. Additionally, organizations must establish strong privacy measures for handling sensitive data, adhering to regulations such as GDPR (General Data Protection Regulation), HIPAA (Health Insurance Portability and Accountability Act), and SOC 2. Disaster Recovery In cloud-based model inference, robust security measures and data protection are essential to prevent data loss and ensure data integrity and availability, particularly for mission-critical applications. Disaster recovery plans should be established to handle potential system failures, data corruption, or cybersecurity threats. Regular data backups, failover mechanisms, and redundancy can mitigate the impact of unforeseen system failures. Popular Tools for ML Model Inference Data scientists, ML engineers, and AI practitioners typically use programming languages like Python and R to build AI systems. Python, in particular, offers a wide range of libraries and frameworks like scikit-learn, PyTorch, Keras, and TensorFlow. Practitioners also employ tools like Docker and Kubernetes to enable the containerization of machine learning tasks. Additionally, APIs (Application Programming Interfaces) play a crucial role in enabling seamless integration of machine learning models into applications and services. There are several popular tools and frameworks available for  model inference in machine learning: Amazon SageMaker: Amazon SageMaker is a fully managed service that simplifies model training and deployment on the Amazon Web Services (AWS) cloud platform. It allows easy integration with popular machine learning frameworks, enabling seamless model inference at scale. TensorFlow Serving: TensorFlow Serving is a dedicated library for deploying TensorFlow models for inference. It supports efficient and scalable serving of machine learning models in production environments. Triton Inference Server: Triton Inference Server, developed by NVIDIA, is an open-source server for deploying machine learning models with GPU support. Check out our curated list of Best Image Annotation Tools for Computer Vision. Model Inference in Machine Learning: Key Takeaways Model inference is a pivotal stage in the machine learning lifecycle. This process ensures that the trained models can be efficiently utilized to process real-time data and generate predictions. Real-time model inference empowers critical applications that demand instant decision-making, such as autonomous vehicles, fraud detection, and healthcare emergencies. It offers a wide array of benefits, revolutionizing decision-making, streamlining operations, and enhancing user experiences across various industries. While model inference brings numerous benefits, it also presents challenges that must be addressed for responsible AI deployment. These challenges include high infrastructure costs, ensuring low latency and interoperability, ethical considerations to avoid biased predictions, model transparency for trust and accountability, etc. Organizations must prioritize ethical AI frameworks, robust disaster recovery plans, continuous monitoring, model retraining, and staying vigilant against inference-level attacks to ensure model accuracy, fairness, and resilience in real-world applications. The future lies in creating a harmonious collaboration between AI and human ingenuity, fostering a more sustainable and innovative world where responsible AI practices unlock the full potential of machine learning inference.

Even as foundation models gain popularity, advancements in object detection models remain significant. YOLO has consistently been the preferred choice in machine learning for object detection. Let’s train the latest iterations of the YOLO series, YOLOv9, and YOLOV8 on a custom dataset and compare their model performance.  In this blog, we will train YOLOv9 and YOLOv8 on the xView3 dataset. The xView3 dataset contains aerial imagery with annotations for maritime object detection, making it an ideal choice for evaluating the robustness and generalization capabilities of object detection models. If you wish to curate and annotate your own dataset for a direct comparison between the two models, you have the option to create the dataset using Encord Annotate. Once annotated, you can seamlessly follow the provided code to train and evaluate both YOLOv9 and YOLOv8 on your custom dataset. Read the Encord Annotate Documentation to get started with your annotation project.   Prerequisites We are going to run our experiment on Google Colab. So if you are doing it on your local system, please bear in mind that the instructions and the code was made to run on Colab Notebook. Make sure you have access to GPU. You can either run the command below or navigate to Edit → Notebook settings → Hardware accelerator, set it to GPU, and the click Save. !nvidia-smi To make it easier to manage datasets, images, and models we create a HOME constant. import os HOME = os.getcwd() print(HOME) Train YOLOv9 on Encord Dataset Install YOLOv9 !git clone https://github.com/SkalskiP/yolov9.git  %cd yolov9 !pip install -r requirements.txt -q !pip install -q roboflow encord av # This is a convenience class that holds the info about Encord projects and makes everything easier. # The class supports bounding boxes and polygons across both images, image groups, and videos. !wget 'https://gist.githubusercontent.com/frederik-encord/e3e469d4062a24589fcab4b816b0d6ec/raw/fa0bfb0f1c47db3497d281bd90dd2b8b471230d9/encord_to_roboflow_v1.py' -O encord_to_roboflow_v1.py Imports from typing import Literal from pathlib import Path from IPython.display import Image import roboflow from encord import EncordUserClient from encord_to_roboflow_v1 import ProjectConverter Data Preparation Set up access to the Encord platform by creating and using an SSH key. # Create ssh-key-path key_path = Path("../colab_key.pub") if not key_path.is_file(): !ssh-keygen -t ed25519 -f ../colab_key -N "" -q key_content = key_path.read_text() We will now retrieve the data from Encord, converting it to the format required by Yolo and storing it on disk. It's important to note that for larger projects, this process may encounter difficulties related to disk space. The converter will automatically split your dataset into training, validation, and testing sets based on the specified sizes.  # Directory for images data_path = Path("../data") data_path.mkdir(exist_ok=True) client = EncordUserClient.create_with_ssh_private_key( Path("../colab_key").resolve().read_text() ) project_hash = "9ca5fc34-d26f-450f-b657-89ccb4fe2027" # xView3 tiny encord_project = client.get_project(project_hash) converter = ProjectConverter( encord_project, data_path, ) dataset_yaml_file = converter.do_it(batch_size=500, splits={"train": 0.5, "val": 0.1, "test": 0.4}) encord_project_title = converter.title Download Model Weight We will download the YOLOv9-e and the gelan-c weights. Although the YOLOv9 paper mentions versions yolov9-s and yolov9-m, it's worth noting that weights for these models are currently unavailable in the YOLOv9 repository. !mkdir -p {HOME}/weights !wget -q https://github.com/WongKinYiu/yolov9/releases/download/v0.1/yolov9-e-converted.pt -O {HOME}/weights/yolov9-e.pt !wget -P {HOME}/weights -q https://github.com/WongKinYiu/yolov9/releases/download/v0.1/gelan-c.pt You can predict and evaluate the results of object detection with the  YOLOv9 weights pre-trained on COCO model.  Check out the blog YOLOv9 Explained and How to Run it if you want to run object detection on pre-trained YOLOv9 weights.  Train Custom YOLOv9 Model for Object Detection We train a custom YOLOv9 model from a pre-trained gelan-c model. !python train.py \ --batch 8 --epochs 20 --img 640 --device 0 --min-items 0 --close-mosaic 15 \ --data $dataset_yaml_file \ --weights {HOME}/weights/gelan-c.pt \ --cfg models/detect/gelan-c.yaml \ --hyp hyp.scratch-high.yaml You can examine and validate your training results. The code for validation and inference with the custom model is available on Colab Notebook. Here we will focus on comparing the model performances. Converting Custom YOLOv9 Model Predictions to Encord Active Format pth = converter.create_encord_json_predictions(get_latest_exp("detect") / "labels", Path.cwd().parent) print(f"Predictions exported to {pth}") Download the predictions on your local computer and upload them via the UI to Encord Active for analysis of your results. Moving on to training YOLOv8! Train YOLOv8 on Encord Dataset Install YOLOv8 !pip install ultralytics==8.0.196 from IPython import display display.clear_output() import ultralytics ultralytics.checks() Dataset Preparation As we are doing a comparative analysis of two models, we will use the same dataset to train YOLOv8. Train Custom YOLOv8 Model for Object Detection from ultralytics import YOLO model = YOLO('yolov8n.pt')  # load a pretrained YOLOv8n detection model model.train(data=dataset_yaml_file.as_posix(), epochs=20)  # train the model model.predict() The code for running inference on the test dataset is available on the Colab Notebook shared below. Converting Custom YOLOv8 Model Predictions to Encord Active Format pth = converter.create_encord_json_predictions(get_latest_exp("detect", ext="predict") / "labels", Path.cwd().parent) print(f"Predictions exported to {pth}") Download this JSON file and upload it to Encord Active via UI. Comparative Analysis on Encord Active On Encord Active under the tab Model Evaluation, you can compare both the model’s predictions.  You can conveniently navigate to the Model Summary tab to view the Mean Average Precision (mAP), Mean Average Recall (mAR), and F1 score for both models. Additionally, you can compare the differences in predictions between YOLOv8 and YOLOv9. Precision  YOLOv8 may excel in correctly identifying objects (high true positive count) but at the risk of also detecting objects that aren't present (high false positive count). On the other hand, YOLOv9 may be more conservative in its detections (lower false positive count) but could potentially miss some instances of objects (higher false negative count).  Recall In terms of recall, YOLOv8 exhibits superior performance with a higher true positive count (101) compared to YOLOv9 (43), indicating its ability to correctly identify more instances of objects present in the dataset. Both models, however, show an equal count of false positives (643), suggesting similar levels of incorrect identifications of non-existent objects. YOLOv8 demonstrates a lower false negative count (1261) compared to YOLOv9 (1315), implying that YOLOv8 misses fewer instances of actual objects, highlighting its advantage in recall performance. Precision-Recall Curve Based on the observed precision-recall curves, it appears that YOLOv8 achieves a higher Area Under the Curve (AUC-PR) value compared to YOLOv9. This indicates that YOLOv8 generally performs better in terms of both precision and recall across different threshold values, capturing a higher proportion of true positives while minimizing false positives more effectively than YOLOv9. Precision-Recall Curve is not the only metric to evaluate the performance of models. There are other metrics like F1 score, IOU distribution, etc.  For more information on different quality metrics, read the blog Data, Label, & Model Quality Metrics in Encord.   Metric Correlation The metric impact on performance in Encord refers to how specific metrics influence the performance of your model. Encord allows you to figure out which metrics have the most influence on your model's performance. This metric tells us whether a positive change in a metric will lead to a positive change (positive correlation) or a negative change (negative correlation) in model performance. The dimensions of the labeled objects significantly influence the performance of both models. This underscores the importance of the size of objects in the dataset. It's possible that the YOLOv9 model's performance is adversely affected by the presence of smaller objects in the dataset, leading to its comparatively poorer performance. Metric Performance The Metric Performance in model evaluation in Encord provides a detailed view of how a specific metric affects the performance of your model. It allows you to understand the relationship between a particular metric and the model's performance. In conclusion, the comparison between YOLOv8 and YOLOv9 on Encord Active highlights distinct performance characteristics in terms of precision and recall. While YOLOv8 excels in correctly identifying objects with a higher true positive count, it also exhibits a higher false positive count, indicating a potential for over-detection. On the other hand, YOLOv9 demonstrates a lower false positive count but may miss some instances of objects due to its higher false negative count. If you want to improve your object detection model, read the blog How to Analyze Failure Modes of Object Detection Models for Debugging for more information.   The precision-recall curve analysis suggests that YOLOv8 generally outperforms YOLOv9, capturing a higher proportion of true positives while minimizing false positives more effectively. However, it's important to consider other metrics like F1 score and IOU distribution for a comprehensive evaluation of model performance. Moreover, understanding the impact of labeled object dimensions and specific metric correlations can provide valuable insights into improving model performance on Encord Active.

For quite some time, the idea that artificial intelligence (AI) could understand visual and textual cues as effectively as humans seemed far-fetched and unimaginable.  However, with the emergence of multimodal AI, we are seeing a revolution where AI can simultaneously comprehend various modalities, such as text, image, speech, facial expressions, physiological gestures, etc., to make sense of the world around us. The ability to process multiple modalities has opened up various avenues for AI applications. One exciting application of multimodal AI is Vision-Language Models (VLMs). These models can process and understand the modalities of language (text) and vision (image) simultaneously to perform advanced vision-language tasks, such as Visual Question Answering (VQA), image captioning, and Text-to-Image search. In this article, you will learn about:  VLM architectures. VLM evaluation strategies. Mainstream datasets used for developing vision-language models. Key challenges, primary applications, and future trends of VLMs. Let’s start by understanding what vision-language models are. 🔥 NEW RELEASE: We released TTI-Eval (text-to-image evaluation), an open-source library for evaluating zero-shot classification models like CLIP and domain-specific ones like BioCLIP against your (or HF) datasets to estimate how well the model will perform. Get started with it on GitHub, and do ⭐️ the repo if it's awesome. 🔥 What Are Vision Language Models? A vision-language model is a fusion of vision and natural language models. It ingests images and their respective textual descriptions as inputs and learns to associate the knowledge from the two modalities. The vision part of the model captures spatial features from the images, while the language model encodes information from the text. The data from both modalities, including detected objects, the spatial layout of the image, and text embeddings, are mapped to each other. For example, if the image contains a bird, the model will learn to associate it with a similar keyword in the text descriptions. This way, the model learns to understand images and transforms the knowledge into natural language (text) and vice versa. Training VLMs Building VLMs involves pre-training foundation models and zero-shot learning. Transfer learning techniques, such as knowledge distillation, can be used to fine-tune the models for more specific downstream tasks. These are simpler techniques that require smaller datasets and less training time while maintaining decent results. Modern frameworks, on the other hand, use various techniques to get better results, such as Contrastive learning. Masked language-image modeling. Encoder-decoder modules with transformers and more. These architectures can learn complex relations between the various modalities and provide state-of-the-art results. Let’s discuss these in detail. Vision Language Models: Architectures and Popular Models Let’s look at some VLM architectures and learning techniques that mainstream models such as CLIP, Flamingo, and VisualBert, among others, use. Contrastive Learning Contrastive learning is a technique that learns data points by understanding their differences. The method computes a similarity score between data instances and aims to minimize contrastive loss. It’s most useful in semi-supervised learning, where only a few labeled samples guide the optimization process to label unseen data points. Contrastive Learning For example, one way to understand what a cat looks like is to compare it to a similar cat image and a dog image. Contrastive learning models learn to distinguish between a cat and a dog by identifying features such as facial structure, body size, and fur. The models can determine which image is closer to the original, called the “anchor,” and predict its class. CLIP is an example of a model that uses contrastive learning by computing the similarity between text and image embeddings using textual and visual encoders. It follows a three-step process to enable zero-shot predictions. Trains a text and image encoder during pretraining to learn the image-text pairs. Converts training dataset classes into captions. Estimates the best caption for the given input image for zero-shot prediction. CLIP Architecture VLMs like CLIP power the semantic search feature within Encord Active. When you log into Encord → Active → Choose a Project → Use the Natural Language search to find items in your dataset with a text description. Here is a way to search with natural language using “White sneakers” as the query term: Read the full guide, ‘How to Use Semantic Search to Curate Images of Products with Encord Active,' in this blog post. ALIGN is another example that uses image and textual encoders to minimize the distance between similar embeddings using a contrastive loss function. PrefixLM PrefixLM is an NLP learning technique mostly used for model pre-training. It inputs a part of the text (a prefix) and learns to predict the next word in the sequence. In Visual Language Models, PrefixLM enables the model to predict the next sequence of words based on an image and its respective prefix text. It leverages a Vision Transformer (ViT) that divides an image into a one-dimensional patch sequence, each representing a local image region. Then, the model applies convolution or linear projection over the processed patches to generate contextualized visual embeddings. For text modality, the model converts the text prefix relative to the patch into a token embedding. The transformer's encoder-decoder blocks receive both visual and token embeddings. It is there that the model learns the relationships between the embeddings. SimVLM is a popular architecture utilizing the PrefixLM learning methodology. It has a simpler Transformer architecture than its predecessors, surpassing their results in various benchmarks. It uses a transformer encoder to learn image-prefix pairs and a transformer decoder to generate an output sequence. The model also demonstrates good generalization and zero-shot learning capabilities. SimVLM Architecture Similarly, VirTex uses a convolutional neural network to extract image features and a textual head with transformers to manage text prefixes. You can train the model end-to-end to predict the correct image captions by feeding image-text pairs to the textual head. VirTex Architecture Frozen PrefixLM While PrefixLM techniques require training visual and textual encoders from scratch, Frozen PrefixLM allows you to use pre-trained networks and only update the parameters of the image encoders. For instance, the architecture below shows how Frozen works using a pre-trained language model and visual encoder. The text encoder can belong to any large language model (LLM), and the visual encoder can also be a pre-trained visual foundation model. You can fine-tune the image encoder so its image representations align with textual embeddings, allowing the model to make better predictions. Frozen Architecture Flamingo's architecture uses a more state-of-the-art (SOTA) approach. It uses a CLIP-like vision encoder and an LLM called Chinchilla. Keeping the LLM fixed lets you train the visual encoder on images interleaved between texts. The visual encoders process the image through a Perceiver Sampler. The technique results in faster inference and makes Flamingo ideal for few-shot learning. Flamingo Architecture Multimodal Fusing with Cross-Attention This method utilizes the encoders of a pre-trained LLM for visual representation learning by adding cross-attention layers. VisualGPT is a primary example that allows quick adaptation of an LLM’s pre-trained encoder weights for visual tasks. VisualGPT Architecture Practitioners extract relevant objects from an image input and feed them to a visual encoder. The resulting visual representations are then fed to a decoder and initialized with weights according to pre-trained LLM. The decoder module balances the visual and textual information through a self-resurrecting activation unit (SRAU). The SRAU method avoids the issue of vanishing gradients, a common problem in deep learning where model weights fail to update due to small gradients. As such, VisualGPT outperforms several baseline models, such as the plain transformer, the Attention-on-Attention (AoA) transformer, and the X-transformer. Masked-language Modeling (MLM) & Image-Text Matching (ITM) MLM works in language models like BERT by masking or hiding a portion of a textual sequence and training the model to predict the missing text. ITM involves predicting whether sentence Y follows sentence X. You can adapt the MLM and ITM techniques for visual tasks. The diagram below illustrates VisualBERT's architecture, trained on the COCO dataset. VisualBERT Architecture It augments the MLM procedure by introducing image sequences and a masked textual description. Based on visual embeddings, the objective is to predict the missing text. Similarly, ITM predicts whether or not a caption matches the image. No Training You can directly use large-scale, pre-trained vision-language models without any fine-tuning. For example, MAGIC and ASIF are training-free frameworks that aim to predict text descriptions that align closely with the input image.  MAGIC uses a specialized score based on CLIP-generated image embeddings to guide language models' output. Using this score, an LLM generates textual embeddings that align closely with the image semantics, enabling the model to perform multimodal tasks in a zero-shot manner. ASIF uses the idea that similar images have similar captions. The model computes the similarities between the training dataset's query and candidate images. Next, it compares the query image embeddings with the text embeddings of the corresponding candidate images. Then, it predicts a description whose embeddings are the most similar to those of the query image, resulting in comparable zero-shot performance to models like CLIP and LiT. ASIF Prediction Strategy Knowledge Distillation This technique involves transferring knowledge from a large, well-trained teacher model to a lighter student model with few parameters. This methodology allows researchers to train VLMs from larger, pre-trained models. For instance, ViLD is a popular VLM developed using the knowledge distillation methodology. The model uses a pre-trained open-vocabulary image classification model as the teacher to train a two-stage detector (student). The model matches textual embeddings from a textual encoder with image embeddings. ViLD Architecture Knowledge distillation transfers knowledge from the image encoder to the backbone model to generate regional embeddings automatically. Only the backbone model generates regional embeddings during inference, and it matches them with unseen textual embeddings. The objective is to draw correct bounding boxes around objects in an image based on textual descriptions. Evaluating Vision Language Models VLM validation involves assessing the quality of the relationships between the image and text data. For an image captioning model, this would mean comparing the generated captions to the ground-truth description. You can use various automated n-gram-based evaluation strategies to compare the predicted labels in terms of accuracy, semantics, and information precision. Below are a few key VLM evaluation metrics. BLEU: The Bilingual Evaluation Understudy (BLEU) metric was originally proposed to evaluate machine translation tasks. It computes the precision of the target text compared to a reference (ground truth) by considering how many words in the candidate sentence appear in the reference.  ROUGE: Recall-Oriented Understudy for Gisting Evaluation (ROUGE) computes recall by considering how many words in the reference sentence appear in the candidate. METEOR: Metric for Evaluation of Translation with Explicit Ordering (METEOR) computes the harmonic mean of precision and recall, giving more weight to recall and multiplying it with a penalty term. The metric is an improvement over others that work with either Precision or Recall, as it combines information from both to give a better evaluation. CIDEr: Consensus-based Image Description Evaluation (CIDEr) compares a target sentence to a set of human sentences by computing the average similarity between reference and target sentences using TF-IDF scores. 🔥 NEW RELEASE: We released TTI-Eval (text-to-image evaluation), an open-source library for evaluating zero-shot classification models like CLIP and domain-specific ones like BioCLIP against your (or HF) datasets to estimate how well the model will perform. Get started with it on GitHub, and do ⭐️ the repo if it's awesome. 🔥 Now that you have learned evaluation metrics pertinent to Vision-Language Models (VLMs), knowing how to curate datasets for these models is essential. A suitable dataset provides fertile ground for training and validating VLMs and is pivotal in determining the models' performance across diverse tasks. Datasets for Vision Language Models Collecting training data for VLMs is more challenging than collecting data for traditional AI models since it involves collecting and quality-assuring multiple data modalities. Below is a list of several datasets combining images and text for multimodal training. LAION-5B: Practitioners use the LAION-5B dataset to build large, pre-trained VLMs. The dataset contains over five billion image-text pairs generated from CLIP, with descriptions in English and foreign languages, catering to a multilingual domain. PMD: The Public Model Dataset (PMD) originally appeared in the FLAVA paper and contains 70 billion image-text pairs. It is a collection of data from other large-scale datasets, such as COCO, Conceptual Captions (CC), RedCaps, etc. This dataset is a reservoir of multimodal data that fosters robust model training. VQA: Experts use the VQA dataset to fine-tune pre-trained VLMs for downstream VQA and visual reasoning tasks. The dataset contains over 200,000 images, with five questions per image, ten ground-truth answers, and three incorrect answers per question. ImageNet: ImageNet contains over 14 million images with annotations categorized according to the WordNet hierarchy. It’s helpful in building models for simple downstream tasks, such as image classification and object recognition. Despite the availability of high-quality multimodal datasets, VLMs can face significant challenges during the model development process. Let’s discuss them below. Limitations of Vision Language Models Although VLMs are powerful in understanding visual and textual modalities to process information, they face three primary challenges: Model complexity. Dataset bias. Evaluation difficulties. Model Complexity Language and vision models are quite complex on their own, and combining the two only worsens the problem. Their complexity raises additional challenges in acquiring powerful computing resources for training, collecting large datasets, and deploying on weak hardware such as IoT devices. Dataset Bias Dataset biases occur when VLMs memorize deep patterns within training and test sets without solving anything. For instance, training a VLM on images curated from the internet can cause the model to memorize specific patterns and not learn the conceptual differences between various images. Evaluation Strategies The evaluation strategies discussed above only compare a candidate sentence with reference sentences. The approach assumes that the reference sentences are the only ground truths. However, a particular image can have several ground-truth descriptions. Although consensus-based metrics like CIDEr account for the issue, using them becomes challenging when consensus is low for particular images. Another challenge is when a generic description applies to several images. Spurious Correlation As the illustration shows, a VLM can annotate or retrieve several relevant images that match the generic caption. However, in reality, the model is nothing more than a bag-of-words. All it’s doing is considering words, such as ‘city,’ ‘bus,’ ‘lights,’ etc., to describe the image instead of actually understanding the caption's sequential order and true contextual meaning. Furthermore, VLMs used for VQA can generate highly confident answers to nonsensical questions. For instance, asking a VLM, “What color is the car?” for an image that contains a white horse will generate the answer as “white” instead of pointing out that there isn’t a car in the picture. Lastly, VLMs lack compositional generalization. This means that their performance decreases when they process novel concepts. For example, a VLM can fail to recognize a yellow horse as a category since it’s rare to associate the color yellow with horses. Despite many development and deployment challenges, researchers and practitioners have made significant progress in adopting VLMs to solve real problems. Let’s discuss them briefly below. Applications of Vision Language Models While most VLMs discussed earlier are helpful in captioning images, their utility extends to various domains that leverage the capability to bridge visual and linguistic modalities. Here are some additional applications: Image Retrieval: Models such as FLAVA help users navigate through image repositories by helping them find relevant photos based on linguistic queries. An e-commerce site is a relevant example. Visitors can describe what they’re looking for in a search bar, and a VLM will show the suitable options on the screen. This application is also popular on smartphones, where users can type in keywords (landscapes, buildings, etc.) to retrieve associated images from the gallery. Generative AI: Image generation through textual prompts is a growing domain where models like DALL-E allow users to create art or photos based on their descriptions. The application is practical in businesses where designers and inventors want to visualize different product ideas. It also helps create content for websites and blogs and aids in storytelling. Segmentation: VLMs like SegGPT help with segmentation tasks such as instance, panoptic, semantic, and others. SegGPT segments an image by understanding user prompts and exploiting a distinct coloring scheme to segment objects in context. For instance, users can ask SegGPT to segment a rainbow from several images, and SegGPT will efficiently annotate all rainbows. [Video] Frederik and Justin discussed how Visual-Language Models (VLMs) power AI in different industries, including their efficiency over Large Language Models (LLMs). Future Research The following are a few crucial future research directions in the VLM domain: Better Datasets The research community is working on building better training and test datasets to help VLMs with compositional understanding. CLEVR is one example of this effort. CLEVR Dataset As the illustration shows, it contains images of novel shapes, colors, and corresponding questions that allow experts to test a VLM’s visual reasoning capacity. Better Evaluation Methods Evaluation challenges warrant in-depth research into better evaluation methods for building more robust VLMs. One alternative is to test VLMs for individual skills through the ARO benchmark. Attribute identification, relational reasoning, and word-order sensitivity (ARO) are three skills that VLMs must master. ARO Dataset The illustration above explains what ARO entails in different contexts. Using such a dataset, experts can analyze what VLMs learn and how to improve the outcomes. 🔥 NEW RELEASE: We released TTI-Eval (text-to-image evaluation), an open-source library for evaluating zero-shot classification models like CLIP and domain-specific ones like BioCLIP against your (or HF) datasets to estimate how well the model will perform. Get started with it on GitHub, and do ⭐️ the repo if it's awesome. 🔥 Robotics Researchers are also using VLMs to build purpose-specific robots. Such robots can help navigate environments, improve warehouse operations in manufacturing by monitoring items, and enhance human-machine interaction by allowing robots to understand human gestures, such as facial expressions, body language, voice tones, etc. Medical VQA VLMs’ ability to annotate images and recognize complex objects can help healthcare professionals with medical diagnoses. For example, they can ask VLMs critical questions about X-rays or MRI scans to determine potential problems early. Vision-Language Models: Key Takeaways Visual language modeling is an evolving field with great promise for the AI industry. Below are a few critical points regarding VLMs: Vision-language models are a multimodal architecture that simultaneously comprehends image and text data modalities. They use CV and NLP models to correlate information (embeddings) from the two modalities. Several VLM architectures exist that aim to relate visual semantics to textual representations. Although users can evaluate VLMs using automated scores, better evaluation strategies are crucial to building more reliable models. VLMs have many industrial use cases, such as robotics, medical diagnoses, chatbots, etc.