Introducing EMG-1: the largest open dataset of wrist EMG synchronized with egocentric video

Human demonstrations are fast becoming a major source of training data for robotic manipulation, especially for learning how people interact with objects during real work.^[1] With egocentric cameras, we can move with a worker through a task, allowing natural object handling to be recorded without constraining the activity to a fixed camera view.

We’ve already made considerable progress in hand-tracking by combining egocentric video with other multimodal inputs and sensor measurements to improve reconstruction. In our Hands at Work article, we show how stereo depth improves the accuracy of 3D hand tracking, while visual odometry and inertial sensors allow us to track movement in world space as the camera moves. Yet new sense modalities remain largely untapped as a means of recovering finger articulation under object occlusion, and measurements like contact or force applied to an object.

Our view is that muscle activity should be part of the data we collect for manipulation. Much of finger movement is controlled by muscles in the forearm, whose electrical activity we can record at the wrist using electromyography (EMG). Those signals continue through a grasp, including when the fingers are hidden. Encord’s interest goes beyond articulation, to whether muscle activity can reveal the timing of contact, changes in grip and the force being applied. EMG is not a direct measurement of joint position, object contact or grip force, so some work is required to make it useful for manipulation; large-scale recordings that pair the signal with video are needed, together with independently measured hand poses against which models can be evaluated.

Encord, in collaboration with Wearable Devices, has developed EMG-1, the largest open dataset pairing wrist EMG with synchronized egocentric video. The field collection comprises 307 hours of real-world work, recorded with three-channel EMG on both wrists alongside head and wrist video, depth, and inertial measurements. The release also includes a complementary 10.4-hour laboratory collection, which uses eight-channel EMG and optical motion capture to provide independently measured finger articulation. The field recordings capture natural object interaction at scale, while the laboratory set provides the ground truth needed to measure the accuracy of EMG-based hand-pose predictions.

Our experiments provide an initial demonstration of what these signals can offer. Reconstruction failures are common: even without objects in the scene, our stereo camera reference failed to produce an acceptable 3D hand pose on 24.7% of evaluated laboratory frames.^[2] Detecting a hand in an image is not the same as recovering a reliable pose label. Wearable Devices’ combined EMG-and-camera model achieved 15.3 mm median fingertip error, approximately 25% lower than our camera/held-pose reference; where that reference failed to reconstruct an acceptable pose, the combined model still produced an estimate, with lower error across every evaluated gap-duration bin. Separate experiments examined EMG-based movement prediction through simulated tracking gaps and recognition of thumb-to-fingertip proximity.

Researchers can use EMG-1 to validate the feasibility of this technology, investigate the timing of grasps and object contact, and explore how muscle activity can provide supervision when visual hand tracking becomes unreliable. We firmly believe there is much more to investigate in these signals than our initial experiments cover, and we’re excited to see what the community can do with EMG-1.

01.EMG collected during real manipulation

There are already substantial datasets for studying hand movement from muscle signals. Meta’s emg2pose contains 370 hours of 16-channel wrist EMG paired with measured hand poses, but does not include synchronized first-person video of real-world object manipulation. EgoEMG combines bilateral wrist EMG with egocentric video over approximately ten hours of recordings. Large egocentric video datasets such as EgoDex provide another source of manipulation data, but without the same synchronized muscle measurements.^[3]

EMG-1 brings these two sources of information together at scale. Unlike controlled hand-motion experiments, these recordings include object handling, natural changes in hand position and occlusions that occur throughout ordinary work; muscle signals continue to be recorded even when the cameras cannot provide a reliable view of the fingers. Optical markers included in the laboratory collection provide an independent measurement of finger articulation, allowing models trained on EMG or video to be evaluated against something other than their own predicted labels.

For both collections, Encord utilized Wearable Devices’ Kavana neural sensing technology to measure EMG activity. The field collection uses Mudra Pro, with three electrodes positioned on the inner side of each wrist, primarily sampling activity from the flexor side of the forearm. The laboratory set uses Mudra Ultimate, with eight electrodes distributed around the wrist circumference to capture a broader range of muscle activity, including flexor and extensor regions. Eight-channel recordings provide greater spatial coverage of muscle activity for experiments against optical motion-capture ground truth.

02.Capturing and synchronizing muscle signals

Recording muscle activity alongside video comes with two main challenges: (a) wrist-worn EMG sensors must maintain skin contact during normal work, and (b) timing must be accurate between sensor streams. Partial or poor electrode contact naturally degrades the EMG recording; likewise timing errors can associate otherwise valid muscle activity with the wrong stage of a grasp or finger movement. With these challenges in mind, we evaluated devices from six manufacturers before selecting Wearable Devices’ wristbands for our collection efforts. Leaving the fingers uncovered, unlike gloves, the wristbands don’t interfere with fine manipulation and are suitable for industrial recording environments.^[4] Their relatively low cost compared with alternative devices also makes large-scale collection much more practical.

We also designed a five-face AprilTag cap for each wristband, giving the cameras a reference for wrist identity and pose as the hands move through the scene. The left and right bands use different tag IDs, which helps distinguish the wrists even when both appear in the same view. We are extending this with the band IMU to carry wrist orientation between tag detections.

Synchronizing the recordings required particular care. Our field EMG data is sampled at 2,133 Hz, while the cameras operate at approximately 30 frames per second; we aligned wristband and wrist-camera recordings by comparing their gyroscope measurements. Both sensors observe the same wrist movements, allowing us to estimate their relative timing without relying on when the recording computer received each signal. The field synchronization check gave a median estimated offset of 0 ms, with qualifying recordings falling within ±10 ms; median correlation between the motion signals was 0.976.^[5]

Our vision pipeline predicts 21 hand joints in 3D, and we check all resulting poses for visibility, geometric consistency and agreement between camera views before accepting them as labels. Where the reconstruction is unreliable, we retain the gap rather than treating an uncertain prediction as a measured pose, with corresponding EMG signals remaining available.

03.Recovering hand pose from EMG and video

Even in the controlled laboratory recordings, without objects obscuring the fingers, our stereo camera reference failed to produce an acceptable 3D hand pose on 24.7% of evaluated frames.^[6] Running MediaPipe to estimate hand poses from both views, we accepted a reconstruction only when both cameras agreed and the resulting joints satisfied geometric consistency checks.

We found that the system could often detect the hand but could not reconstruct its finger positions consistently across the two cameras; most failures were not caused by the hand disappearing from view. Of the 24.7% failure rate, 20.6 percentage points came from reconstructed joints failing those checks, 3.4 from inconsistent pairing between views, and just 0.7 from missing hand detections.

Our laboratory optical motion-capture recordings allow us to measure how well alternative approaches recover finger articulation, including during these camera-reference failures. To this end, Wearable Devices developed a model combining eight-channel EMG with a separate camera-based pose estimator. The model uses an EMG encoder to learn features from muscle activity, which are combined with visual information to predict finger joint angles. To test generalization across wearers, the model was trained on other participants before receiving two minutes of calibration data from each person being evaluated.

For the camera reference, we held the last accepted pose during tracking gaps and measured the error against optical motion capture. Dropping these frames would make the camera reference look better by removing precisely the failures we wanted to measure. Across seven participants, 125 takes and approximately 349,000 evaluated frames, the combined model achieved 15.3 mm median fingertip error, compared with 20.3 mm for the camera/held-pose reference, a reduction of approximately 25%. The difference is particularly interesting when the camera reference loses track of the fingers. Figure 2 shows how their errors change with the duration of the interruption.

The fused model maintains substantially lower error as the camera reference becomes unavailable. Within the first half-second of a tracking gap, the held-pose error rises to 26.0 mm, while the combined model achieves 16.8 mm. This advantage persists across the longer intervals, although these are less common: 99% of naturally occurring reference gaps in the laboratory recordings last less than 0.61 seconds.

We supplied both methods with true palm and finger-base positions from motion capture for this evaluation; the reported errors concern finger articulation rather than wrist localization. The fused model also uses half a second of future EMG, making this an offline annotation result rather than a demonstration of real-time tracking. And because Wearable Devices’ model has its own camera branch, this comparison does not isolate how much of the improvement comes from EMG. It does, however, establish that the combined system produces more accurate finger-articulation estimates than the defined stereo camera reference, including during periods when that reference cannot reconstruct an acceptable pose.

04.What muscle signals tell us beyond hand pose

The combined model improves on the camera reference, but how much information can we recover from EMG itself? To investigate this, we tested whether muscle signals could predict finger movement during simulated gaps in visual tracking, without access to new camera observations.

Starting from a known hand pose, we compared three approaches over interruptions lasting between half a second and four seconds: holding the last pose; predicting movement from previous poses using a learned model; and tracking finger articulation from EMG. The test comprised 86 simulated gaps across four held-out trials from a single participant. Holding the last pose is a reasonable baseline for a short interruption, but over longer gaps we also wanted to test EMG against a model capable of predicting movement from the preceding poses.

Holding the last pose works best for the shortest interruption. As the gap lengthens, however, EMG becomes more useful. At four seconds, its mean fingertip error is 33.0 mm, compared with 42.6 mm for holding the last pose and 38.6 mm for the learned pose-only model. EMG outperforms the pose-only model at every tested duration. The experiment is limited in scale but provides evidence that muscle activity contains information about subsequent finger movement beyond what can be predicted from pose history.

Recognizing finger proximity from muscle activity

With no manipulated objects in the laboratory trials, we began with a proximity measure we could obtain directly from the optical markers—thumb-to-fingertip distance—and tested whether EMG could recognize when it fell below 20 mm. This is a measurable proxy for a pinch-like movement, rather than a label for actual object contact. Using amplitude and frequency features extracted from 200 ms EMG windows, we trained a simple logistic-regression classifier and evaluated it on participants excluded from training.

Across all ten participants and 643 takes, the classifier achieved AUROC 0.74 (95% confidence interval 0.70–0.78) and average precision 0.27, against a positive-label frequency of 0.11. With only 11% of windows labeled positive, a random-ranking baseline would have an average precision of approximately 0.11. Individual participant AUROC values ranged from 0.63 to 0.83. This suggests that even relatively simple EMG features carry information about finger-to-finger proximity across wearers. Establishing whether those signals can reliably identify actual object contact, grip changes or release will require annotated interactions with objects. The field recordings provide a basis for investigating those questions.

05.Releasing EMG-1 for research

The experiments presented here explore only a small part of what can be learned from muscle activity recorded alongside human movement. With EMG-1, we’re making the underlying recordings available so that researchers can develop and test their own approaches. Researchers can work directly with the original sensor measurements rather than relying on predictions from our models.

One unresolved question is how well models developed on controlled hand movement generalize to real object interactions, particularly across different conditions and electrode configurations. We’re also interested in how much information three electrodes on the inner wrist can recover, and what an eight-channel band adds when tested under the same conditions.

The datasets will be released on Hugging Face with their raw sensor streams, camera-derived annotations, quality metadata and laboratory motion-capture ground truth. Fused and EMG-only pose estimates, continuous tag-derived wrist poses, and object-contact annotations and force estimates are planned for subsequent releases.

Our next collection will extend eight-channel EMG into real working environments with the full head-and-wrist camera rig. We also plan to evaluate generalization across operators, sessions and electrode placements, alongside object contact and wrist tracking.

EMG-1 is a starting point for this work. Video remains an essential source of information about human manipulation, but it does not capture everything involved in producing movement. By making muscle activity available alongside visual and motion measurements, we hope to give researchers another way to study the physical actions that robots need to learn.

06.Notes and References

1. Tao, T. et al., “DexWild: Dexterous Human Interactions for In-the-Wild Robot Policies,” Robotics: Science and Systems (2025). doi:10.15607/RSS.2025.XXI.075.
2. Compared against MediaPipe.
3. Salter, S. et al., “emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation,” NeurIPS Datasets and Benchmarks (2024); Xi, Z. et al., “EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation,” arXiv:2605.05712 (2026); Hoque, R. et al., “EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video,” ICLR (2026).
4. Roda-Sales, A. et al., “Effect on manual skills of wearing instrumented gloves during manipulation,” Journal of Biomechanics 98 (2020), 109512, doi:10.1016/j.jbiomech.2019.109512; Dianat, I., C. M. Haslegrave, and A. W. Stedmon, “Methodology for evaluating gloves in relation to the effects on hand performance capabilities: a literature review,” Ergonomics 55(11) (2012), 1429–1451, doi:10.1080/00140139.2012.708058.
5. The check covered 4,799 qualifying wristband-side recordings from 2,612 field clips, with gyroscope correlation of at least 0.8. Offsets were evaluated on a 10 ms grid; 97% of scorable pairs met the correlation threshold, with median correlation 0.976. A finer check on 45 clips measured offsets between −11 and +1 ms.
6. Evaluation covers approximately 707,000 stereo frame steps from 245 laboratory takes across nine participants. MediaPipe Hand Landmarker was run on both views of the fixed stereo camera; a pose was accepted only when the hand could be matched across views and the reconstructed joints passed the geometric consistency checks, including a 3-pixel reprojection threshold.