Data Curation for Robotics: Finding Failure Modes Before They Cost You a Deployment
Robotic Learning Lead at Encord
TL;DR: Most robot failures that show up in deployment were already visible in the training data, they just weren't found in time. Data curation for robotics is the practice of selecting, balancing, and correcting the demonstrations and sensor streams data a robot learns from, specifically to catch failure modes like distribution shift, strategy imbalance, and rare edge cases before they reach the production stage. This guide covers why curation matters for robotics specifically, the common failure modes it needs to catch, the key steps involved, where curation applies across robotics applications, and how to measure whether it's actually working.
Imagine that a robot arm successfully passes every benchmark in the lab. However, three weeks after deployment, it starts dropping objects on a specific shelf, in a specific lighting condition, at a specific time of day. Nothing in the code changed. The model wasn't retrained. What changed is that the robot is now seeing something its training data never really covered.
This is the pattern behind most robot failures that make it to production: it's not a bug causing failure, but a gap in the data the model trained on. The fix isn't more data in general, teams that just collect more of the same end up with bigger datasets and the same blind spots. The fix is Data Curation: deliberately finding, and correcting for, the failure modes hiding in a dataset before a robot finds them for you in the field.
Why Data Curation matters for robotics
In most software, a bug is a bug; you can trace it to a line of code. In a robot running a learned policy, the "bug" often lives in the training data, not the code path. So when a model fails on a task it hasn't seen enough of, or fails inconsistently because its training data taught it two conflicting ways to do the same thing, there's no stack trace pointing at the problem. It looks like a modeling failure, but in reality It's actually a data failure.
That distinction matters more in robotics than almost anywhere else in AI, for 3 key reasons:
- Failures are physical, not just statistical: A misclassified image is a bad prediction. A robot that fails to release a grip, or mistimes a stop, is a dropped object, a damaged product, or a safety incident. The cost of an uncaught failure mode is measured in downtime and hardware, not just accuracy points.
- Collection is expensive, so waste is expensive: Unlike text or web images, robotics data has to be physically generated with a robot, an operator, and an environment, every time. Curating badly means paying twice: once to collect data that didn't need collecting, and again to collect the data that was actually missing.
- Deployment conditions never match the lab exactly: Lighting changes, objects shift position, floors get slightly more reflective. A dataset that looked one way in training and validation can still miss the specific long-tail conditions a robot might incur on day one in the field.
How Data Curation for Robotics differs from other domains
The core curation lifecycle of sourcing, cleaning, structuring, balancing, and monitoring data is similar across every AI domain. What changes for robotics specifically is what has to be balanced and how failure shows up.
For Computer vision, curation focuses on image and video balance. NLP curation focuses on deduplication and quality filtering at web scale. Robotics curation adds two things neither of those deal with: multiple sensor streams that have to stay aligned in time (a camera frame, a LiDAR sweep, and a force reading all describing the same instant), and action sequences, where what matters isn't just what the robot saw, but what it did next, and whether that was the right thing to do.
That combination, multimodal alignment plus temporal action structure, is what makes robotics failure modes distinct enough to warrant their own taxonomy.
An example of robotics training data
Common failure points in robotics data
Not every failure mode looks the same, and curation methods differ depending on which one you're trying to catch. The table below breaks down the most common categories.
| Failure mode | How It Shows Up | Root Cause in the Data |
| Distribution shift | Model behaves unpredictably on inputs that differ from training conditions (new lighting, new object, new floor surface) | The dataset looks complete on paper, but only covers a narrow slice of what the robot actually meets in deployment. |
| Strategy imbalance | Multiple valid ways to complete a task exist in the demonstrations (e.g., left-hand-first vs. right-hand-first), and the model blends them unpredictably at inference | Both strategies "look correct" individually, so standard quality checks don't flag the ambiguity |
| Rare events and near-misses | Recoveries, close calls, and edge-case object interactions are underrepresented relative to routine successful runs | Manual review of large-scale logs doesn't scale, so rare frames never get opened |
| Sensor and perception drift | Model accuracy degrades slowly in production as sensors recalibrate, degrade, or encounter new conditions over time | Drift is invisible in a static, one-time training set; it only shows up by comparing training data to live production data over time |
| Annotation-induced errors | Inconsistent labels across a temporal sequence (an object loses its tracked identity, an action boundary is mislabeled) | These errors are specific to the labeling process itself, not the raw sensor data, so they need review focused on annotation consistency, not just data quality |
This taxonomy matters because failure isn't one thing, treating it as a single bucket hides the patterns that actually need fixing. Gupta, Ciftci and Bansal (2025) demonstrated a method that turns this from a manual bottleneck into something scalable: instead of a person scrolling through thousands of failure clips one at a time, a vision-language model reads each failure and generates a structured explanation of what went wrong, then similar explanations get automatically grouped into clusters. The result is a small number of clearly named categories, say, 'drops objects on reflective surfaces' or 'stalls on tight turns', pulled out of what would otherwise be an unsorted pile of individual incidents.
Key steps in Curation for robotics data
The general curation lifecycle (source, clean, structure, balance, monitor) still applies. For robotics specifically, 3 steps do most of the work in catching the failure modes above.
1. Multimodal and temporal alignment: Before anything else, camera, LiDAR, depth, and force streams need to be synchronized to the same clock, and action sequences need consistent time stamping across the whole trajectory. Curation decisions made on misaligned data are decisions made on the wrong data.
2. Action sequence balancing: Robotics curation has to check that the distribution of strategies in a task is balanced, not just that images are varied enough.
- Strategy detection: if 90% of demonstrations solve a task one way and 10% solve it another way, that split needs to be visible before training, not discovered after.
- Blended behavior risk: a model trained on an unbalanced mix of strategies can blend both approaches unpredictably at inference, producing behavior that matches neither demonstration cleanly.
- Rebalancing over volume: the fix is usually adjusting the mix of what already exists, not collecting more data on top of an imbalance data that's still there.
3. Failure mode surfacing. This is where quantitative methods matter most.
Influence functions. These use the math behind how a model trains to work out which specific demonstrations are causing a policy to succeed or fail, so engineers can trace a bad test rollout back to the exact training samples responsible, instead of guessing.
- Data attribution: they estimate how much a model's behavior would change if one specific demonstration were upweighted, downweighted, or removed entirely.
- Causal linking: they connect a failed rollout directly to the training sequence that caused it, rather than treating failure as unexplainable.
- No extra labels needed: they work directly on the policy's training dynamics, so they don't require any additional human annotation to run.
- Targeted curation: the output is a ranked list of demonstrations, so curation becomes a targeted edit (remove the ones causing harm, keep or duplicate the ones driving performance) rather than a blind rebalance.
Agia et al. (2025) built this into CUPID, and showed that training on under a third of a curated dataset matched full-dataset performance on a standard manipulation benchmark. Xu et al. (2026) extended the approach with ATHENA, making it computationally practical at the scale of billion-parameter vision-language-action models trained across dozens of tasks at once.
Semantic clustering of failure logs. Instead of comparing failures by how similar the raw footage looks, this groups them by underlying cause.
- Pattern grouping: a model reads each failure and writes a short explanation of what went wrong, then similar explanations get clustered together.
- Named categories: the output is a small set of clearly labeled patterns, for example "drops objects on reflective surfaces," instead of an unsorted pile of individual clips.
- Scale without manual review: this turns a job that doesn't scale (someone scrolling through thousands of clips) into something structured and repeatable.
Active learning. This decides which examples are actually worth a human's time to review, rather than treating every frame in a dataset as equally important.
- Uncertainty targeting: it flags the cases a model is least confident about, since those are the most likely to contain a genuine gap.
- Review efficiency: it points reviewers at a small, high value subset instead of asking them to work through a dataset frame by frame.
4. Validation before it goes back into training. A curated batch still needs a quality pass: a duplicate check, a label consistency check, and confirmation that the balance changes actually shifted the distribution the way intended, before it's trusted for retraining.
Applications of curation in robotics data
Curation methods aren't one-size-fits-all across robotics. What counts as a failure mode, and what curation needs to catch, changes by application.
| Application | What curation focuses on | Typical failure mode to catch |
| Manipulation and grasping | Balancing grasp strategies, contact points, and approach angles across demonstrations | Strategy imbalance; ambiguous multi-modal action distributions |
| Locomotion | Coverage of terrain types, recovery behaviors after a stumble or slip | Underrepresented recovery sequences; distribution shift on new surfaces |
| Humanoid whole-body control | Coordination across many joints and contact points simultaneously, often across multiple embodiments | Cross-limb strategy conflicts; data sparsity for full-body coordination tasks |
| Autonomous navigation and perception | Sensor fusion across camera, LiDAR, and radar; rare object and obstacle coverage | Sensor drift; rare-object and edge-case detection gaps |
| Human-robot interaction (HRI) | Balanced coverage of operator commands, corrections, and unexpected human behavior | Near-miss and safety-relevant interaction underrepresentation |
From detection to correction
Finding a failure mode is only half the job. The next step is deciding what to do with that finding, and the two most common mistakes go in opposite directions:
- Blanket re-collection. Collecting significantly more data across the board, hoping volume fixes a specific gap. This is expensive and often doesn't fix the underlying imbalance at all.
- Ignoring the fix once it's found. Teams sometimes catalog failure modes without feeding them back into a targeted collection or rebalancing plan, so the same issue resurfaces in the next model version.
The more reliable approach is targeted correction: use the specific failure mode identified (a terrain type, a strategy imbalance, a lighting condition) to guide exactly what gets collected or rebalanced next, then re-validate against the same evaluation the failure was originally caught on,not just a general benchmark.
Measuring whether curation is working
Curation isn't done once a batch is rebalanced. Teams need a way to check that it actually improved something.
| Metric | What it Informs |
| Retention balance across tasks/strategies | Whether rebalancing actually corrected the imbalance, or shifted it somewhere else |
| Success-rate delta, pre/post curation | Whether the curated dataset measurably improves closed-loop task performance, not just dataset-level statistics |
| Edge-case coverage | Whether previously rare conditions (a terrain type, a lighting condition) now have adequate representation |
| Failure recurrence rate | Whether a previously identified failure mode is showing up less often in new deployment logs |
Encord Built for Robotics Data Curation
Curating robotics data specifically requires handling multimodal sensor alignment, action-sequence balance, and failure-mode surfacing in one connected workflow, rather than as separate manual steps. Encord's data curation platform supports embeddings-based and natural-language search across camera, LiDAR, depth, and time-series data in one interface, with automated quality metrics that surface duplicates, class imbalance, and outliers before data reaches annotation.
For teams specifically working on Physical AI and robotics data, Encord connects curation directly to annotation and model evaluation, so a failure mode identified during evaluation routes straight back into the next collection or curation pass,closing the loop between what a robot gets wrong in the field and what its next training set actually contains.
Key takeaways
- Most robot failures in deployment trace back to gaps in training data, not the code,curation is how those gaps get caught before they become production incidents.
- Robotics curation adds multimodal sensor alignment and action-sequence balancing on top of the standard curation lifecycle.
- Common failure modes include distribution shift, strategy imbalance, rare events, sensor drift, and annotation inconsistency,each needs a different detection approach.
- Influence-function methods and semantic failure clustering are emerging as the most effective ways to surface failure modes at scale, rather than relying on manual log review.
- Curation applies differently across manipulation, locomotion, humanoid control, navigation, and HRI,the failure modes that matter shift by application.
- Detection only matters if it's followed by targeted correction and re-validation, not blanket re-collection.
Explore more
Frequently asked questions
A failure mode is a specific, recurring pattern in how a robot's training data fails to represent a real-world condition, causing predictable errors in deployment,for example, underrepresented recovery behaviors, conflicting demonstration strategies, or sensor drift between training and production environments.
Curation decides what data is worth using and how it should be balanced before and after labeling. Annotation adds the ground-truth labels,action boundaries, object tracks, grasp labels,to the data curation has already selected.
Yes, to a meaningful extent. Methods like influence-function attribution and evaluation-rollout analysis can surface failure modes in offline testing, before a robot ever reaches production. They won't catch every possible real-world condition, but they materially reduce how many failures are discovered for the first time in the field.
No. Curation determines what to prioritize collecting next and rebalances what already exists,it doesn't eliminate the need for real-world data, especially for genuinely novel conditions no existing dataset covers.