VLMs and Egocentric Understanding: Keeping Score

On 1,500 egocentric video questions, human annotators still beat the best VLMs, and majority voting helps humans but not models.

Category :
Applied Research & Automation
Author :
Martin Fisch & Felix Cohen, ML Team
Date :
November 11, 2026
Read time :
6 min
Key findings
Key findingsDetail
Humans still beat VLMs8 points for a single annotator (35% fewer errors) and 13 points for a panel, against the current leader GPT-6 Astra. The gap is larger in some categories.
Human consensus extends the lead significantlyConsensus seeking improves results for humans but not for VLMs. Human individuality provides valuable decorrelation.
VLM collaboration is neutral or destructiveAutomated pipelines which rely on multiple VLMs are likely wasting tokens.

“Action without vision is a nightmare” - Japanese Proverb

Vision Language Models (VLMs) form the backbone and starting-point for many robotic action pipelines, known as Vision Language Action (VLA) models. Yet despite the amazing progress foundation models have made in the last few years, they still trail human understanding in spatial awareness and subtle action understanding.

We often work with the leading robotics companies to caption videos for VLA pre-training. The typical video involves a human or robot protagonist, filmed from an egocentric camera view, interacting with objects and the scene around them. Typically, coarse captions describe a 5–30 second sequence, while fine-level captions cover 2–4 seconds of action. Caption styles and levels of detail vary sharply from customer to customer.

Providing the right action verb, object and hand involved is table stakes. Beyond that, we often have to describe the object, provide color or material, distinguish the object spatially (“third bowl from the left”), place the locative (“from the middle shelf beneath the kitchen counter”), describe the ego motion in a variety of ways (“walks across the room passing between the fireplace and the sofa”), rate action quality and discuss future intent.

These captions can then be used to train a better VLM, or as pre-training data for a VLA model. But while VLMs are very good at spewing sensible-seeming text approximately related to the scene at hand, the gaps in their actual understanding translate to errors and hallucinations which can be fatal for physical AI. To better identify where VLMs add value, we have taken a rigorous approach evaluating each release and assessing their relative strengths and weaknesses.

One of the more surprising results was how answers combine. Human consensus improved accuracy, while combining VLMs provided no benefit despite using more tokens. The familiar wisdom-of-crowds effect seems to apply to humans, but not to VLMs.

01.Encord’s Egocentric VQA Benchmark (1500)

We created a bank of 1500 human-curated video questions and answers built from private, real-world data across different VLA training scenes: household, industrial, and countertop. We created question groups on different aspects of scene understanding, such as object detail, action attributes, temporal and spatial reasoning. All of the questions are derived from the types of actual captions and scenes we provide to robotics customers. This ensures the questions cover facts that robotics labs really need as signals.

We use a private dataset of unpublished videos to ensure the data is truly unseen. The academic datasets may have been scraped or indirectly absorbed through broad-based training regimes, so we prefer to use unseen footage for fair benchmarking across frontier labs. Ground truth was pulled from previously reviewed labels and then vetted a second time by a research team member. To make scoring procedural, objective and repeatable for each new model we settled for multiple choice, except for object identification which is open-ended and judged procedurally by object hierarchy rules. Many of these questions have competing choices where two answers are valid but one is better. Real life is not multiple choice, so this should be a low bar on the way to physical autonomy.

To provide a baseline, we also had a few professional annotators take the same test. It was somewhat unfair as they only saw two-second clips from scattered scenes with no continuity. Typically annotators would see longer segments and, for now, humans are better at utilizing longer horizon continuity than most VLMs. We did not do a round of quality review, and we did not control for human tiredness. For replacing humans in a monotonous industrial task that may be a fair comparison, but for a household task this is probably harsh on the human performance level.

Example 1

Where are the bowls in relation to nearby objects?

  1. A.On counter
  2. B.Near stove
  3. C.Sink(correct answer)
  4. D.By faucet

Result

All three annotators chose C. Two of three Astra runs chose A.
Scene recreated for illustration.

Example 2

Where is the hat in relation to nearby objects?

  1. A.Behind box(correct answer)
  2. B.On top of box
  3. C.In front of box
  4. D.Left of box

Result

All three annotators chose A. Two of three Astra runs chose D.
Scene recreated for illustration.

02.Results

We show how benchmarked VLMs have performed over time (Figure 1) and include the results from the latest leaderboard (Table 1) as of October 2026.

Model progress versus the human panel

  • Proprietary (size undisclosed)
  • Open weights, sized by parameters
  • Best open model so far
  • Best proprietary so far
  • Humans: majority of 3
  • Human annotator

Accuracy on the 1,500 questions (%)

Model progress versus the human panel
ModelRelease dateAccuracy (%)Weights
Kimi-VL-A3BApr 202523.07 (95% CI 20.93 to 25.21)Open, 16B parameters
InternVL3-78BApr 202558.2 (95% CI 55.7 to 60.7)Open, 78B parameters
InternVL3-38BApr 202555.53 (95% CI 53 to 58.06)Open, 38B parameters
InternVL3-8BApr 202551.42 (95% CI 48.87 to 53.97)Open, 8B parameters
Qwen3-VL-4BOct 202555.67 (95% CI 53.15 to 58.18)Open, 4B parameters
Qwen3-VL-8BOct 202550.07 (95% CI 47.54 to 52.6)Open, 8B parameters
Molmo2-8BDec 202552.82 (95% CI 50.28 to 55.35)Open, 8B parameters
Gemini 3.1 ProFeb 202666.2 (95% CI 63.81 to 68.59)Proprietary
PhysBrain (LIBERO)Apr 202653.53 (95% CI 51.01 to 56.06)Open, 4B parameters
Qwen3.6-27BApr 202655.64 (95% CI 53.1 to 58.18)Open, 27B parameters
LLaVA-OneVision2-8BMay 202654.67 (95% CI 52.15 to 57.19)Open, 8B parameters
Cosmos3-SuperMay 202666.8 (95% CI 64.42 to 69.18)Open, 32B parameters
InternVideo3-8BJun 202643.6 (95% CI 41.09 to 46.11)Open, 8B parameters
GPT-5.6 LunaJul 202652.33 (95% CI 49.81 to 54.86)Proprietary
Gemini 3.7 FlashAug 202673.27 (95% CI 71.03 to 75.51)Proprietary
Qwen3.8-27BAug 202667.4 (95% CI 65.03 to 69.77)Open, 27B parameters
Gemini 3.8 FlashSept 202673 (95% CI 70.75 to 75.25)Proprietary
GPT-6 AstraSept 202676.51 (95% CI 74.37 to 78.66)Proprietary
Figure 1: Accuracy on our 1,500-question benchmark by model release date.
Table 1: Encord Egocentric VQA (1,500 questions) leaderboard as of October 2026.

The key results:

  • Humans still beat VLMs. In aggregate the gap is 8 points for a single annotator (or 35% fewer errors) versus GPT-6 Astra, the current leader, but the gap is larger in some categories.
  • Location and spatial awareness plus counting remain major VLM weaknesses:
    • Relative location (−22 points): locating an object in relation to another nearby object.
    • Precise location (−13 points): where exactly an object sits, is taken from or placed, on a surface or within a container.
    • Object presence (−12 points): does the object exist in the scene, as a check for hallucinations.
    • Counting (−20 to −30 points): how many of a potentially numerous object or class of objects exists in the scene.
  • VLMs are better than humans at pure physical attributes like color and material.
  • Human consensus is additive. 2-of-3 consensus seeking improves results for humans (+5 points, or +13 versus best VLM).
  • VLM collaboration adds no value (0 points), whether through mixing families or re-running at different temperatures.
  • Pairing a human + VLM is a pragmatic middle ground (+2.4 points). Here we bring in a second human as a decider for disagreements (30% of the total), and this mode reflects some current annotation workflows.
  • The frontier lead holds. Paywalled frontier lab models like Astra are 9 points ahead of the open-source pack, although the latter are much lighter weight models available for fine-tuning.
  • We will probably need some harder questions for 2027, as the frontier models have really closed the gap this year.

03.Wisdom of crowds is a human trait that does not extend to VLMs

An important result of our study, confirmed by recent external research on different tasks, shows that while majority voting improves results for humans, the same approach adds little-to-no value for VLMs.

Humans draw on idiosyncratic background knowledge, life experiences, viewpoints, perception biases, education levels, nationalities and biological differences. For our best-of-three panel we used annotators from similar backgrounds yet still reduced errors by one-third compared to individual scores. A wider or more diverse panel would likely further improve the collective accuracy.

Creating a panel of VLMs just burned tokens and degraded performance versus the best single VLM. Using the best VLMs with multiple runs and temperature settings only drew even, yet again at a much higher cost. This is unsurprising as modern VLMs share common vision backbones, aligned training mixtures, pre-training pipelines with shared data, common instruction-tuning regimes and similar RL post-training. Model errors are correlated rather than independent.

While not as performant as a full human panel, a hybrid combination of human annotator plus VLM, with an extra human only dragged in for dispute resolution, does improve upon the single human results, likely with reduced cost or staffing requirements. This is a practical validation for model-assisted human input.

04.Conclusion

These results have real implications for automated pipelines which claim to validate results through multi-VLM queries. Unless this approach can demonstrate real value in the specific domain, we suggest saving tokens and just using the best model available.

To achieve the highest level of accuracy, consensus voting among humans is currently the best by a significant margin. Due to the higher cost of additional annotators, in practice this approach may suit building the golden-set or the most demanding use-cases. This also suggests that a well managed human quality control stage and disagreement resolution process can play an important role in boosting the overall accuracy of a data set by virtue of naturally introducing a second opinion.

Finally, this is only a multiple-choice exam. Getting to human levels is only one step in the journey to AI alignment. Freeform caption answers are considerably more challenging and leave more scope for VLM hallucinations. Nevertheless, from our own research and trials we find that relative leaderboard positioning correlates well with downstream model performance on more complex egocentric tasks.

05.References

  1. Bugaud, Z. (2026). Hidden Clones: Exposing and Fixing Family Bias in Vision-Language Model Ensembles. arXiv preprint arXiv:2603.17111.
  2. Tong, Y., Hou, Y., Cui, S., Bosselut, A., & Sachan, M. (2026). Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models. arXiv preprint arXiv:2605.30713.

Get access to EMG-1

The largest open dataset of wrist EMG synced to egocentric video. Sign up and you’ll get access to the full 300+ hour dataset and 8-channel sample right away.

Subscribe for new releases

Research at the frontier.