Hands at Work: Egocentric hand pose tracking for real world scenes

Understanding human motion and interactions is one of the cornerstones of computer vision. Using human action videos as a training and transfer learning tool for humanoid robotics has recently kindled strong industry interest in accurate hand tracking, particularly from an egocentric perspective and for videos in the wild.

Egocentric videos introduce challenges for accurate hand tracking. The hands often drop out of view when they are lowered to hip level, the head can pivot away from the hands and, unlike in exocentric views, the association of hand with protagonist versus bystander can sometimes be challenging, particularly at the individual frame level.

In-the-wild footage adds further challenges. For data capture in studios, the protagonist is usually instructed to constrain their motion for hand visibility, head motion is slow and hands are usually kept in crisp view. In the wild, the protagonist engages in more natural motion where the head often pivots rapidly, adding motion blur, and the hands can swing in and out of view at speed.

Additionally, many of the use cases for robotics are industrial in nature, a real shift compared to the historical consumer VR/AR impetus for much of the earlier research in the field. Industrial workers often wear thick gloves for health and safety, and these are harder to track as most models are trained only for bare-hand prediction.

Data collection for robotics companies has also become multimodal by demand. Stereo depth, IMU and additional wrist sensors create an opportunity for a more holistic approach to hand tracking. For instance, our pipeline can use stereo depth input to further reduce tracking error, already at state-of-the-art accuracy.

The combination of these factors amounts to a domain shift which leaves many existing hand tracking models ill-suited for current use cases. Most published metrics ignore these problems, by assuming away bystanders and hand swaps, reporting on bare-hand datasets only and using clean studio video with slow-moving hands. The models we compare against are all exceptional models at their design tasks, and some provided the foundations we have built our own pipeline around. However, in our experience their designs are not adequate for handling many real-world egocentric challenges. In this article we review each of these challenges, discuss their consequences and sketch out how we approached them with our own model, Encord-REACH (Robust Ego Aware Cross-modal Hands), demonstrating they can be resolved with high fidelity through a computer vision pipeline.

01.Ego hands vs bystander hands are a serious confound

0:00 / 0:00

0:00 / 0:00

0:00 / 0:00

0:00 / 0:00

Swapping hands between the ego protagonist and a bystander is a real error condition which is ignored by most public metrics. For downstream use cases, swapping people can be catastrophic for action planning and for tracking real-world hand trajectories. Picture a co-worker at the same bench reaching into the shared work area from just right of the wearer’s own right hand:

• both hands wear the same company gloves;
• both are at a similar distance and scale;
• both enter the frame from the lower edge;
• both are oriented with the fingers pointing away from the camera.

In any one frame the two hands look interchangeable. What separates them is spread over time: where each hand came from, and whether its forearm stays connected to the wearer’s body edge; how it moves with the camera’s own motion; whether it moves in coordination with the wearer’s other hand; what it was doing seconds before it reached the shared area.

The majority of academic 3D hand datasets, the basis for much of the published hand pose research, typically include only a single subject in view. Notable exceptions include the large web-video training sets of WiLoR^[15] and WildHands^[9], and COCO-WholeBody^[4], but none of these label “ego” vs “other”. This has led to entire families of models which offer no mechanism for associating and tracking hands to the ego throughout a video.

To quantify and correct this issue we collected our own dataset of in-the-wild egocentric scenes, using human annotation to train and validate approaches to this problem. Many of the videos in our set have nearby bystanders entering the frame from all angles.

We found that off-the-shelf models and methods were insufficiently capable of distinguishing ego from bystander in harder cases. Most recent egocentric video 3D hand methods work in two steps. First they detect and reconstruct hands frame by frame. They then fit or refine a trajectory over those per-frame results. Our tests demonstrated that the second step cannot reliably repair swapping failures in the first. Even if initially tracked properly, as soon as the ego protagonist’s hands drop off camera many models eagerly fit the corresponding hand from the nearest bystander.

To quantify this in a fair manner, we used EgoHands^[1], a niche but open dataset well suited to this issue. The dataset’s scenarios are considerably easier than the one we outlined above, as the bystander is always facing across from the camera, but helpfully much of the footage has bystander hands. Yet even in this simple case, the typical ego-dedicated hand approach swaps to the bystander for 20–50% of frames, and more general models (such as MediaPipe^[3]), with no trained concept of “ego”, do so over 70% of the time.

When prompted properly for the task, a leading LLM like Astra^[20] does quite well on EgoHands but much poorer on our private dataset, hallucinating almost half of the absent hands. For the EgoHands dataset we cannot eliminate the possibility that it has ingested the ground truth, as the set is public and scrapable. The gap between performance on the public and private test sets makes us cautious about Astra’s reliability as a detector for demanding real-world data.

Following our exploration of the model space in search of the optimal foundation, we firmly believe the best approaches are built on video-native models rather than frame-based ones (irrespective of post-processing). Video models can look across longer horizons and benefit from video continuity cues. Switching to a video approach allowed our model to reliably differentiate the ego hands and, in particular, to avoid jumping to bystanders when the ego hand dropped out of view. Training on a difficult ego-vs-bystander dataset helped our model master the difference.

Our model reliably detects only the ego wearer’s hands, with few hallucinations, misses and bystander swaps: 0% swaps on an easier dataset like EgoHands and 0.5% on our more challenging in-house test set.

02.Gloves are common in factories but rare in datasets

0:00 / 0:00

0:00 / 0:00

Some of the biggest opportunities in humanoid robotics are in industrial workplaces. Ego video in this environment often involves thick safety gloves. This adds a significant challenge for hand tracking and hand pose detection, as most research and datasets focus on bare hands. Gloves remove many of the fine-grained texture signals, such as knuckles and creases, that vision models use to pin down the hand pose. Different colours, patterns, materials and sizes all add confounding signals, causing most off-the-shelf models to degrade considerably. Even when provided the cropped gloves (and many models fail to even identify the glove in the scene), many models perform poorly, as Figure 1 shows.

Ground truth

Human-annotated

Ours

NME 8.2

Astra

NME 19

MediaPipe

NME 23.8

RTMPose

NME 25.1

ViTPose

NME 27.2

In our view this is partly a data problem and partly an architecture problem. We built a dataset with varied gloved hands in use, interacting with objects and environments in the real world. On our held-out glove set, our pipeline’s normalized mean error is less than half that of the nearest open baselines, and shows similar outperformance against most academic models. While gloved predictions remain somewhat noisier and less accurate than bare-hand ones, our model now delivers reliable and credible predictions throughout the scene.

From an architecture perspective, RTMPose^[6] outperforming ViTPose^[5] demonstrates that a much lighter convnet can beat a heavy vision transformer thanks to very different supervision strategies. RTMPose uses SimCC supervision, which decomposes localization into 1D axis projections, making coordinate prediction less dependent on local pixel texture (which is different for gloves) and more dependent on overall geometric orientation. RTMPose’s training recipe also uses strong CutOut/Random Erasing and color jittering during fine-tuning, and thus learns to infer hand joints from structural content rather than relying on skin-color shortcuts. Our own model’s architecture and supervision strategy has been carefully optimized to robustly handle these challenges.

03.Multimodal input can be an extension for better tracking

As the physical AI world is increasingly multimodal, we find ourselves working with data across a wide range of modalities:

• Ego video
• Stereo depth
• Visual odometry
• Head IMU and accelerometer
• Wrist cameras
• Wrist IMU
• EMG

Outputs include camera-space joint positions, world-space joint positions, depth-map-fit joint positions, wrist camera footage and grasp force metrics (from EMG only).

Our approach has been to build a modular and flexible pipeline which can accommodate different information flows while providing consistent annotations. The core of our pipeline is detecting, tracking and predicting 3D hand poses from only monocular ego video, with state-of-the-art accuracy for keypoints and wrist tracking in camera space. A strong base model benefits every variation. Adding stereo depth to the video stream allows us to further improve our base model performance, as our pipeline can use the depth map to resolve depth ambiguities and reduce tracking error. Most of the adjustment is in the overall hand position rather than finger articulation, as stereo depth is less accurate for small objects like fingers. In Table 1 we demonstrate how incorporating depth improves our pipeline on TACO^[11], an open dataset which provides depth maps along with the hand-object interaction. Incorporating depth has a particularly strong impact on the tail of the distribution, by resolving the most uncertain pose predictions.

For some use cases, the focus switches to world-space accuracy where, instead of tracking the hand relative to the camera view, we track in 3D space, usually in a gravity-aligned reference frame. Picture a hand resting on a table in front of the ego protagonist while the head pivots and looks around the room. In camera space the hand is constantly moving. In world space it is stationary.

The high-quality data capture rigs Encord uses provide accurate head camera motion and rotation, usually through a combination of visual odometry and head IMUs. This allows us to map our camera predictions directly to world space with high fidelity. In other cases we use classic computer vision and deep learning methods to calculate these mappings directly from the ego video. Depending on environmental features this can be noisier, but it still allows a good understanding of world-frame dynamics.

Wrist cameras and wrist IMUs are yet another potential source of input data. Often the wrist cameras offer only a very limited view of finger articulation, but these can further help track the wrists and improve overall tracking accuracy, as well as provide important visual cues of object interactions for downstream robotics models.

Finally, at Encord we are constantly innovating. In a related project we are working with specialist providers to track even completely occluded grasps via EMG signals through an additional wearable. We will showcase this technology in a separate paper.

04.The Encord hand rig is not license gated

In order to offer customers annotations free of licensing requirements, we developed the Encord hand rig based on common conventions. Our hand model is a 21-joint keypoint 3D skeleton defined by visible anatomical landmarks (the COCO-WholeBody convention^[4]), rather than MANO’s^[2] mesh-regressed rotation centers. Our 3D labels are triangulated from calibrated multi-view keypoints, with per-subject bone lengths regularized toward population proportions, so the skeleton carries no mesh, skinning or PCA shape prior. We then verify the fitting optically for select frames per dataset with human review. Due to licensing restrictions on the published MANO annotations, for each of the datasets below we measure our own results based on fidelity to the Encord hand rig.

Compared and measured on the HO-Cap dataset^[12], which provides fitted keypoints solved via a MANO rig, our skeleton shifts several joints systematically, although usually within around 10mm of the corresponding MANO joint. Our wrist is typically 12mm further toward the forearm. Our finger knuckles (MCP) are typically 6mm closer to the wrist and 5mm closer to the surface. Off-the-shelf models trained on MANO would likely report higher losses on our dataset, and vice versa for our model on MANO-annotated datasets, although with minor fine-tuning of the head they would likely adapt, as the systematic differences mostly follow bone-aligned offsets.

For reporting evaluation we use the OakInk2 dataset^[10], which comprises tabletop scenes with hand-object interactions and occasionally involves off-camera hands. We reference this dataset because (a) independently published metrics are available for most modern hand pose research approaches, and (b) the authors made this dataset public with a license suitable for commercial reporting.

05.Benchmarking our 3D accuracy

The two core 3D metrics for hand pose estimation are Camera Translation (CT), which expresses the wrist position error in metric units, and Root-Relative Mean Per Joint Position Error (RR-MPJPE), which expresses the finger articulation error relative to the wrist. Various iterations extend these for camera-space MPJPE, Procrustes-fit MPJPE, world-space MPJPE and so forth. Our hand pose pipeline is at state-of-the-art levels on these metrics, but we do not think they are sufficient guidance for real-world use cases, and we prefer to use stricter metrics than generally reported in the literature.

Single-image hand pose research conventionally reports MPJPE and CT with the model given the ground-truth box or crop for each hand. A missed hand cannot happen, because detection is outside the evaluation. Similarly, video and world methods typically report their metrics over frames where the method produced a track for the hand, so frames with no detection are dropped. This naturally skips hard hands and flatters results. Particularly in the context of the challenges we outlined earlier in the paper, and the tracking required for downstream tasks, a more thorough approach is warranted.

A few recent papers, such as Ace-Ego-Hand^[18] and ViDiHand^[17], advocate the “penalized” protocol, CT-p and MPJPE-p, which charges missed detections an effectively fixed penalty. Low recall from poor tracking is given a cost, and this avoids rewarding models which only predict easy hands. The stronger baseline models typically reach 23–30mm MPJPE on the hands they find, but under the -p protocol they spread to 25–55mm. For CT the difference is starker, with 50–80mm on found hands but 85–180mm under the -p protocol.

Yet even this stricter metric is only one side of the correction coin, because it rewards models which always predict a pose. This optimizes for recall at the cost of precision and can manifest as high amounts of hallucination (see Table 2). Notably, the papers advocating this metric are built on generative backbones, which have a proclivity to provide a hand prediction regardless of detection or even the existence of hands. For many robotics tasks, hallucination is a severe flaw.

In our assessments we use a “penalized for precision and recall errors” protocol, -pr, giving equal weight to misses and hallucinations. For egocentric videos in realistic scenarios, the hand is often off camera, and downstream users need to know a hand is absent rather than rely on a hallucinated pose. Just as any -p metric is stricter than its base metric, -pr is stricter than -p. Yet in most cases, our -pr figures are better than the reported looser base metrics of other approaches.

Our results in Table 3 demonstrate the strength of our approach. The metrics across the board are helped by OakInk2 being a much easier dataset than the real-world situations our pipeline has been designed to handle. And, as we demonstrated in the earlier sections, our model is also robust to bystander swaps and can track gloves, unlike many of the baselines.

Finally, our production pipeline can also produce world-space joint estimates, but as the accuracy is heavily impacted by the accuracy of the camera tracking approach, we focus here on the camera-space numbers. Our data collection typically includes accurate visual odometry and IMUs for effective “ground truth” camera-to-world transforms, and when these are absent the tracking depends on the SLAM or VO routine used and the distinctive features in the specific environment.

06.Conclusion

The recent surge in egocentric data collection has created a real need for accurate and robust hand tracking techniques. Despite the multiple confounds that make capturing accurate hand tracking difficult in real-world scenarios, our Encord-REACH pipeline overcomes the challenges with an end-to-end solution already in production for some of the leading robotics companies.

07.References

1. EgoHands: Bambach et al., ICCV 2015.
2. MANO: Romero et al., SIGGRAPH Asia 2017.
3. MediaPipe Hands: Zhang et al., 2020.
4. COCO-WholeBody: Jin et al., ECCV 2020.
5. ViTPose: Xu et al., NeurIPS 2022.
6. RTMPose: Jiang et al., 2023.
7. InterWild: Moon, CVPR 2023.
8. HaMeR: Pavlakos et al., CVPR 2024.
9. WildHands: Prakash et al., ECCV 2024.
10. OakInk2: Zhan et al., CVPR 2024.
11. TACO: Liu et al., CVPR 2024.
12. HO-Cap: Wang et al., NeurIPS 2025.
13. Dyn-HaMR: Yu et al., CVPR 2025.
14. HaWoR: Zhang et al., CVPR 2025.
15. WiLoR: Potamias et al., CVPR 2025.
16. EgoForce: Millerdurai et al., SIGGRAPH 2026.
17. ViDiHand: Wang et al., arXiv 2026.
18. Ace-Ego-Hand: Liu et al., arXiv 2026.
19. EgoFound3R: Fu et al., arXiv 2026.
20. GPT-6 Astra: OpenAI, 2026.