What Are Micro-Models? Encord's Semi-Supervised Annotation Approach for Physical AI Data

TL;DR: Micro-models are small, task-specific models trained on a tiny labeled seed set to automate annotation, using semi-supervised learning to propagate labels across much larger unlabeled datasets. This matters most for Physical AI, where multi-sensor robot data (RGB, LiDAR, depth, proprioceptive) must stay calibrated and consistent, get it wrong and it causes deployment failures and safety risk. Encord applies this through 3D Scenes, cross-sensor calibration, and model-assisted pre-labeling.

Annotating training data for robots and physical AI systems is one of the hardest, most expensive parts of building them. Multiple sensors, high data volumes, and safety-critical accuracy requirements make manual labeling slow and error-prone. This article breaks down what micro-models are, why semi-supervised annotation works so well for this kind of data, where Physical AI annotation gets genuinely difficult, and how Encord's platform applies the micro-model approach across multi-sensor robot data.

What Are Micro-Models?

A micro-model is a small machine learning model trained on a narrow, purpose-selected slice of labeled data to automate one specific annotation task, detecting a single object class, tracking a gripper state, or segmenting a particular surface. Rather than building one model that tries to handle an entire annotation job, teams train many micro-models and chain them together, each one handling a distinct piece of the ontology.

Encord pioneered this approach as an alternative to two bottlenecks in annotation: fully manual labeling (which doesn't scale) and one large general-purpose model (which needs a lot of labeled data before it performs well on any specific task).

How Do Micro-Models Differ From Foundation Models?

1. Scale and resource usage

• Foundation models: Large, general-purpose architectures pretrained on massive, diverse datasets, often requiring significant compute before they perform well on a specific task.
• Micro-models: Small, task-specific models trained on a tiny labeled seed set, sometimes as little as 1–5% of a dataset. They're fast and cheap to train because they're solving one narrow problem, not because they're built for lightweight inference.

1. Scope and versatility

• Foundation models: Built for broad generalization, one model handling many tasks or object classes across a wide range of inputs.
• Micro-models: Deliberately narrow and overfit, one model per feature, object, or action. They're not designed to generalize beyond that task, which is exactly what lets them work well with so little training data.

1. Role in the annotation pipeline

• Foundation models: Useful as a general starting point for pre-labeling common, well-represented classes, but tend to struggle on novel or narrowly defined features they weren't trained to recognize.
• Micro-models: Act as modular, chainable components. Teams train many of them, one per task, and stitch them together to automate a full annotation ontology, with humans reviewing only the lowest-confidence outputs.

How Micro-Models Work

Micro-models rely on 3 core characteristics that distinguish them from typical model training:

1. Narrow focus: Each model handles one specific job, such as spotting a single object class or a single action boundary, rather than trying to generalize across an entire scene.
2. Intentional overfitting: The model is purposely overfit to a small, highly specific pool of labeled data, prioritizing precision on one feature over broad accuracy.
3. Low bias and high speed: Because the task is narrow, training runs quickly, often within an active learning loop that takes minutes rather than hours.

The Semi-Supervised Annotation Loop

The workflow behind a micro-model follows a consistent loop of 5 key steps:

1. Bootstrap with a small labeled set. A handful of well-chosen, representative examples are labeled by hand for one specific task.
2. Generate pseudo-labels. The micro-model scans the remaining unlabeled data and assigns labels to items it recognizes with high confidence.
3. Stitch or ensemble. Multiple task-specific micro-models are combined, like an assembly line, to cover a full annotation ontology.
4. Route low-confidence items for review. Human annotators focus their time on the outputs the model is least confident about, rather than reviewing every frame equally.
5. Retrain on corrected labels. The micro-model is updated with the corrections, improving accuracy on the next pass.

This loop is a direct application of semi-supervised learning, a small amount of labeled data combined with a much larger pool of unlabeled data, rather than requiring every example to be hand-labeled before training starts.

Why Semi-Supervised Annotation, Specifically?

Fully supervised annotation, where annotators label every example by hand before a model ever sees the data, doesn't hold up at the volume most annotation pipelines run at today. Fully unsupervised approaches, on the other hand, give teams no way to steer a model toward the specific features they actually need labeled.

Semi-supervised learning sits between the two, and its efficiency is well documented in the research literature:

• A 2025 multicenter MRI study on brain metastases segmentation found that semi-supervised models trained on half the labeled data matched or outperformed fully supervised models trained on twice as much data, with the largest gains appearing on external test sets (Ottesen et al., 2025).
• Research on low-label settings has trained competitive image classifiers from as few as ten labeled examples per class, by bootstrapping self-supervised features and carefully controlling the information flow between labeled and unlabeled data during training (Rebuffi et al., 2020).

The mechanism these studies describe, using a small labeled set to generate labels across a much larger unlabeled pool, is exactly what a micro-model does at the level of a single annotation task.

Why Micro-Models Matter for Physical AI?

Physical AI systems, robots, autonomous vehicles, and drones, learn from data that is inherently more expensive to collect and label than a typical image dataset. There simply isn't a large, pre-labeled bank of robot manipulation footage sitting around the way there is for, say, street photography.

The Data Scarcity Problem in Robot Learning

Robot learning teams have responded to this scarcity in a few different ways, and none of them fully solve the underlying annotation problem:

• Massive human-video corpora substitute abundant egocentric footage for scarce robot demonstration data.
• Synthetic data engines generate training data without needing a physical robot at all.
• Cheaper teleoperation setups make it more affordable to collect real robot demonstrations at scale.

Individually, these approaches are genuinely useful, but each one adds to the annotation problem rather than solving it. Human video is action-free and unstructured until someone segments, grounds, and aligns it, and the absence of task-aligned annotation is repeatedly cited in the research as the actual blocker to progress, not the volume of raw footage itself. This is precisely the kind of narrow, well-defined labeling task, segment this action, ground this object, align this frame, that a micro-model can be trained on with a small seed set and then applied at scale.

The Real Complexity: Why Robot and Physical AI Data Is Harder to Annotate

Most annotation problems that computer vision solved over the past decade assumed one modality at a time. Such as bounding boxes on photos and segmentation masks on video ect. However, Physical AI data breaks that assumption.

The Multimodal Data Problem: A single robot demonstration typically produces several time-aligned data streams at once such as:

• RGB video from one or more onboard or external cameras
• Depth or RGB-D data for spatial understanding
• LiDAR or radar point clouds for 3D perception in navigation and manipulation
• Proprioceptive data, joint angles, torque, and force feedback from the robot's own actuators
• Egocentric human demonstration video, first-person footage used to teach manipulation policies before a robot ever executes the task
• IMU data for orientation and motion, especially in drones and mobile robots

Each stream needs to be labeled for different things: objects in RGB, surfaces in point clouds, contact events in force data, hand-object interactions in egocentric video, but a model learning to act in the physical world needs all of them to be aligned with each other.

Sensor Fusion Is a Calibration Problem Before It's a Labeling Problem

Camera-LiDAR-radar fusion requires precise temporal and spatial calibration before a single label can be drawn. A LiDAR scan and a camera frame captured milliseconds apart can show the same object in slightly different positions if sensors run at different capture rates or sit in different physical positions on the robot.

This calibration challenge shows up across different Physical AI models such as:

• Humanoid robots fuse RGB cameras with depth sensors.
• Drones fuse cameras with IMU and GPS.
• Industrial robots fuse vision with force-torque sensing.

The modalities differ, but the underlying calibration and consistency problem is the same in every case.

Egocentric and Demonstration Video Has Its Own Labeling Burden

Robot learning increasingly leans on egocentric human video to work around the scarcity of real robot demonstration data. But that footage is unstructured and action-free until it's annotated. A survey of vision-language-action research found that raw egocentric video still needs to be segmented, grounded, and aligned before it can train anything at all, regardless of how it was collected.

The scale of this cost is significant on its own: one widely cited 50,000-clip egocentric dataset built specifically for robot manipulation required roughly 1,500 hours of in-person recording and curation before policy training could even begin, and that's before hand pose, action segments, and object contact are annotated within that footage.

Why Annotation Cost Scales With Sensor Count, Not Dataset Size

A team labeling single-camera image data has one modality to get right. A team labeling a humanoid robot's manipulation demonstrations has to correctly annotate and cross-check five or six data types simultaneously, keep them time-synchronized, and do it at a volume where manual labeling per stream doesn't scale.

Why Physical AI Annotation Accuracy Matters, and What Happens When You Get It Wrong

Cross-Sensor Label Disagreement and Model Failure

If a LiDAR label and a camera label disagree on where an object sits, the model learns conflicting information. That conflict doesn't stay contained to training metrics; it shows up as behaviour in deployment.

Downstream Consequences

Getting Physical AI annotation wrong has consequences that compound the further downstream they travel:

• Deployment failures: a robot that misjudges object position or contact state because its training data had inconsistent cross-sensor labels.
• Safety risk: in autonomous vehicles and industrial robotics specifically, annotation errors around object detection or contact events translate directly into physical safety incidents, not just lower accuracy scores.
• Wasted training cycles: teams often don't discover a labeling inconsistency until a model underperforms in evaluation, at which point the cost of re-annotating and retraining is far higher than catching the error during the labeling process itself.

How Encord Applies Micro-Models to Multimodal Physical AI Data

This is where the theory becomes a working pipeline. Encord's Physical AI suite is built around specific capabilities that make the micro-model approach work across multiple sensor types at once, rather than one modality in isolation.

1. 3D Scenes and Time-Synchronized Sensor Binding

Encord's 3D Scenes bind multiple timestamped sensor streams, point clouds, images, and camera-parameter streams, into one time-synchronized unit, with world, ego, and sensor frames of reference. This makes it possible to propagate a label learned in one modality, such as an object detected in RGB, across paired LiDAR or depth frames, instead of re-labelling each sensor stream from scratch.

1. Calibration Support

Camera intrinsics and extrinsics are supported across distortion models, including pinhole, radial, Brown-Conrady, fisheye, and OpenCV rational-polynomial. Calibration between sensors is handled at the platform level rather than left to each annotator to reconcile manually.

1. Cross-Sensor Object Linking and Temporal Alignment

Cross-sensor object linking and temporal alignment tools keep labels consistent across camera, LiDAR, and radar views inside one unified workspace, directly addressing the calibration-before-labeling problem described above.

1. Model-Assisted Pre-Labeling in Practice

This is where the micro-model methodology plugs in directly: a narrow model trained on a small labeled seed set pre-labels the rest of a data stream, and annotators shift from labeling to reviewing and correcting.

1. Curation and Evaluation

• Encord Curation: Curates and surfaces high-signal or edge-case examples across massive multi-sensor datasets, so review effort focuses on the frames that actually need human judgment.
• Encord post-training-Alignment: Evaluates model performance against ground truth, closing the loop between annotation quality and downstream model behaviour.

The Pipeline End-to-End

1. Ingest raw, multi-sensor robot data RGB, depth, LiDAR, radar, proprioceptive logs) into a unified 3D Scene.
2. Label a small seed set for one specific task, such as a gripper state or an object class.
3. Train a micro-model on that seed set and propagate pre-labels across the rest of the unlabeled stream.
4. Propagate consistent labels across paired sensor modalities using calibrated cross-sensor linking, rather than repeating the labeling process per sensor.
5. Route the lowest-confidence frames to human reviewers first, using curation tooling to prioritize review effort.
6. Retrain the micro-model on corrected labels and chain it with other task-specific micro-models to build out a full annotation ontology.

💡See how this works on your own multi-sensor data ➡️ Book a demo of Encord's Physical AI suite

Key Takeaways

• Micro-models are small, deliberately overfit models trained on a narrow slice of labeled data to automate one annotation task at a time.
• They rely on semi-supervised learning, a small labeled seed set plus a much larger unlabeled pool, an approach research consistently shows can match or beat fully supervised training on far less labeled data (Ottesen et al., 2025; Rebuffi et al., 2020).
• Physical AI data is harder to annotate than typical computer vision data because it spans multiple sensor types that must stay calibrated and consistent with each other, not just individually accurate.
• Getting this wrong has real consequences: cross-sensor label disagreement leads directly to deployment failures, safety risk, and wasted training cycles.
• Encord applies the micro-model approach across multimodal robot data through 3D Scenes, calibration support, cross-sensor linking, model-assisted pre-labeling, and curation and evaluation tooling.

Related Reading

• Physical AI data pipeline guide
• Sensor Fusion Data Annotation guide
• Complete Guide to LiDAR and Point Cloud Annotation
• Top 6 data collection services for Robotics and Embodied AI