Multi-Object Tracking (MOT)

What Is Multi-Object Tracking (MOT)?

Multi-Object Tracking (MOT) is a computer vision task that detects multiple objects in a video and assigns each one a consistent identity across frames, so a system knows not just what is in the scene, but which specific instance it's looking at from one moment to the next.

Unlike single-object tracking, where a model follows one target through a video, Multi-Object Tracking (MOT) must handle a variable number of objects appearing, disappearing, occluding one another, and re-entering the frame, all while keeping their identities intact.

This is what powers use cases like counting pedestrians crossing a street, tracking every vehicle in a traffic scene, or following multiple people through a warehouse camera network without losing or swapping their IDs.

Source: Datature

Why Multi-Object Tracking matters for Physical AI

Multi-Object Tracking (MOT) sits at the core of almost every Physical AI system that has to reason about a dynamic environment over time:

• Autonomous vehicles and ADAS: tracking surrounding vehicles, cyclists, and pedestrians across frames to predict trajectories and avoid collisions

• Robotics: following multiple humans or objects in a workspace to plan safe motion paths

• Warehouse and logistics: tracking pallets, forklifts, and workers across camera feeds for safety and throughput analytics

• Sports and broadcast analytics: tracking every player and the ball across a match

• Smart cities and surveillance:monitoring crowd flow, vehicle counts, and anomaly detection across multi-camera networks

• Retail analytics:tracking shoppers to understand movement patterns and dwell time

In each case, detecting objects frame-by-frame isn't enough, the system needs temporal identity to generate meaningful downstream signals like speed, trajectory, dwell time, or behavior over time.

How Multi-Object Tracking works

Most modern MOT systems follow a tracking-by-detection paradigm, which breaks the problem into two stages:

1. Detection : an object detector (e.g., YOLO, Faster R-CNN, DETR-based models) identifies all objects of interest in each individual frame, producing bounding boxes or masks.
2. Association: a tracking algorithm links detections across consecutive frames to the same object identity, typically using a combination of:

• Motion prediction: (e.g., Kalman filters) to estimate where an object should appear next
• Appearance features: (e.g., re-identification embeddings) to match objects by visual similarity
• Spatial overlap: (e.g., IoU matching) between predicted and detected boxes
• Data association algorithms: (e.g., the Hungarian algorithm) to solve the optimal matching between tracks and new detections

Common MOT algorithms and frameworks

• SORT (Simple Online and Realtime Tracking): a lightweight baseline combining Kalman filtering with IoU-based matching
• DeepSORT : extends SORT with a learned appearance embedding for more robust re-identification through occlusion
• ByteTrack: associates both high- and low-confidence detections, improving tracking through partial occlusions and crowded scenes
• OC-SORT / StrongSORT / BoT-SORT : newer variants that improve motion modeling and camera-motion compensation
• Transformer-based trackers (e.g., TransTrack, MOTR) : treat detection and association jointly within an end-to-end architecture, rather than as two separate stages

Key challenges in Multi-Object Tracking

• Occlusion : objects temporarily blocked from view need to be re-identified correctly once they reappear, without being assigned a new ID
• ID switches: when two objects cross paths or move similarly, trackers can mistakenly swap their identities
• Crowded or dense scenes: high object density increases the difficulty of correct association
• Camera motion: moving or multi-camera setups complicate consistent tracking across viewpoints
• Scale and appearance variation: objects changing size, pose, or lighting across frames

Multi-Object Tracking and training data

Because MOT models depend on both accurate per-frame detection and temporally consistent identity labels, they require video annotation workflows that go beyond static image labeling.

This typically means:

• Frame-by-frame or interpolated bounding box / segmentation annotation
• Persistent object ID labeling across the full video sequence
• Occlusion and re-entry handling in the labeling workflow itself
• Quality control for ID consistency, not just per-frame label accuracy

This is where purpose-built video annotation tools matter: labeling MOT training data manually, frame by frame, doesn't scale, and general-purpose image annotation tools aren't built to preserve object identity across thousands of frames. Encord's video annotation platform supports object tracking natively, including interpolation, re-identification after occlusion, and automated pre-labeling, to help teams build the high-quality, temporally consistent datasets that MOT models need.

{{Training_data_CTA::Annotate, Manage, and Curate Data at Scale for Warehouse Automation Systems with Encord}}