Immerse yourself in vision

DINOv3 is Meta AI’s third generation of open-source self-supervised vision foundation models. It is a 7-billion parameter Vision Transformer trained on 1.7 billion images without labels. The model provides high-quality global and dense features. This can be applied to tasks such as image classification, semantic segmentation, depth estimation, and object tracking.  Source: DINOv3 The significance of DINOv3 lies in three factors: scale, stability, and versatility. It introduces new algorithms to stabilize dense features and releases a family of distilled variants. This way, DINOv3 establishes itself as a general-purpose vision backbone. It reduces reliance on labeled datasets and offers reusable representations that perform well across domains. What’s New in DINOv3? Large Scale Training DINOv3 is trained at a scale that sets it apart from earlier versions. The largest model has 7 billion parameters and was trained on 1.7 billion images, all without human labels. This scale allows the machine learning model to learn visual representations that generalize across a wide range of downstream tasks. The training relies on Vision Transformers (ViTs), which scale effectively with larger datasets and longer training runs. Unlike earlier DINO versions that were limited by instability in dense feature learning, DINOv3 introduces state-of-the-art mechanisms such as Gram Anchoring to maintain stable training at scale. Gram Anchoring In computer vision (CV), dense features refer to patch-level representations that preserve fine-grained details across the entire image. These are essential for tasks like segmentation, depth estimation, and object tracking, where pixel- or region-level accuracy matters. Unlike global features, which summarize an image into a single embedding for classification, dense features must stay consistent and discriminative across long training runs. High-resolution dense features of the image. Source: DINOv3. A key challenge in scaling DINOv3 was that while global representations kept improving, dense features degraded over time. Patches that should remain distinct started to collapse into similar embeddings, hurting performance on dense prediction tasks. To address this, the DINOv3 team introduced Gram Anchoring, a regularization technique that stabilizes dense features during long training. The idea is simple: instead of directly constraining individual patch features, Gram Anchoring works on the Gram matrix, which encodes pairwise similarities between patches. The student model’s Gram matrix is encouraged to stay close to that of an earlier, more stable teacher network (the “Gram teacher”). This approach allows local features to evolve freely, as long as their relative structure remains consistent. Applied after ~1M iterations, Gram Anchoring quickly “repairs” degraded local features and significantly improves dense-task benchmarks like segmentation and depth estimation, while global performance remains strong. Qualitative effect of gram anchoring. Source: DINOv3. The image above shows how dense features improve with gram anchoring. Without it (middle row), features are scattered and noisy. With anchoring (bottom row), they become sharper and more consistent, making objects like flowers, birds, and food stand out more clearly. Universal Frozen Backbone DINOv3 is trained as a multi-purpose frozen backbone, meaning the core model is kept fixed after pretraining. This universal representation works across a wide range of tasks like detection, segmentation, and retrieval without retraining from scratch. The idea is simple: if the backbone is strong and general enough, downstream models can plug into it with minimal fine-tuning, saving compute and preserving stability. Post-hoc Adaptability Even though the backbone stays frozen, DINOv3 enables post-hoc adaptation through lightweight modules. Instead of retraining the full model, researchers can add task-specific heads like linear probes or adapters that specialize the features for new problems. This flexibility makes it easier to apply DINOv3 across domains from natural images to medical or satellite data without redoing the heavy pretraining. Distilled Variants To make DINOv3 more accessible, the Meta AI research team also trained distilled variants. These are smaller student models distilled from the large backbone, keeping much of the performance while reducing size and latency. This opens the door to real-world use cases like robotics or mobile applications where compute is limited but high-quality representations are still needed. How Dinov3 Works At its core, DINOv3 learns dense visual features in a fully self-supervised way. Instead of relying on labeled data, it uses a teacher–student training setup: The teacher model provides target representations. The student model learns to match them using multiple augmented views of the same image. This approach encourages the model to capture rich, invariant features that remain consistent across transformations like cropping, color shifts, or blurs. Over time, the student becomes a strong encoder that produces representations useful for downstream tasks. Training pipeline: curated unlabeled data → large-scale SSL pretraining → Gram anchoring → high-res refinement → distillation into multiple sizes. Source: DINOv3: Self-supervised learning for vision at unprecedented scale. Once pretrained, DINOv3 acts as a universal visual encoder. During inference, images are passed through the backbone ViT to produce dense feature maps. These representations can then be: Adapted directly for tasks like retrieval or clustering. Fine-tuned with lightweight heads for segmentation, detection, or classification. Transferred across domains with minimal supervision, since the features are invariant and robust. Inference pipeline: input → frozen DINOv3 → shared features → lightweight adapters → task-specific outputs. Source: DINOv3: Self-supervised learning for vision at unprecedented scale. How DINOv3 Compares to Previous Models DINOv3 builds on earlier self-supervised methods like MoCo, BYOL, and DINOv2 but pushes three key improvements: Dense patch-level learning: Unlike global embedding models, it supervises at the patch level, enabling strong performance on dense tasks like segmentation. Scalability: Trained with larger Vision Transformers on billions of images, achieving robustness across diverse domains. Universality: Produces features competitive with supervised models, reducing reliance on labeled data. In benchmarks, DINOv3 consistently narrows or surpasses the performance gap between supervised and self-supervised approaches making it one of the most general-purpose visual encoders available today. For more information on previous model, read the blog DINOv2: Self-supervised Learning Model Explained DINOv3 Benchmark Performance DINOv3 consistently matches or surpasses supervised methods across standard vision benchmarks. ImageNet Classification Without seeing a single human-provided label, DINOv3 reaches top-1 accuracy comparable to fully supervised baselines. This is significant because ImageNet has traditionally been the gold standard for supervised learning. Segmentation and Dense Prediction Thanks to patch-level alignment, DINOv3 excels at tasks like semantic segmentation (ADE20K) and object detection (COCO). Earlier SSL models often struggled here, but dense supervision helps DINOv3 preserve fine-grained details. Transfer Learning Across Domains Features pretrained with DINOv3 adapt effectively to very different domains such as medical imaging or satellite data where labeled datasets are scarce. This shows that the model has truly learned universal visual features rather than overfitting to natural images. Scaling Trends As the backbone size grows (ViT-S → ViT-B → ViT-L → ViT-H), downstream performance scales smoothly. Larger DINOv3 models deliver gains in both accuracy and robustness, reinforcing the value of large-scale pretraining. Why this matters: DINOv3 isn’t just a research milestone, it’s a practical backbone replacement. Instead of relying on separate models like ResNet for classification, YOLO for detection, or Mask R-CNN for segmentation, you can use a single self-supervised backbone,i.e., DINOv3 with task-specific heads. This unification means you only need to maintain one backbone across pipelines. In practice, this also opens up new workflows. If you store backbone features from your production environment, you can quickly re-scan past data to find relevant samples for retraining or validation without running full inference pipelines again. Source: DINOv3: Self-supervised learning for vision at unprecedented scale. DINOv3 Real-World Applications DINOv3 is already being deployed in diverse, high-impact domains. World Resources Institute (WRI): WRI uses DINO to measure tree canopy heights from satellite imagery. This helps track global reforestation progress and provides civil society groups with accurate, scalable tools for monitoring environmental change without the need for massive annotated datasets. NASA Jet Propulsion Laboratory (JPL): NASA JPL integrates DINO into Mars exploration robots, enabling them to handle multiple vision tasks like terrain mapping and object recognition with minimal compute resources. This makes autonomous navigation more reliable in extreme environments where human intervention is impossible. These use cases highlight DINO’s versatility: it can adapt to domains with scarce labels like satellite imagery or resource-constrained settings like space robotics, proving its value as a universal vision foundation model. DINOv3 Access and Availability Meta has made DINOv3 openly available to the research community, releasing pretrained weights, documentation, and training details. The models can be accessed directly through: Facebook Research’s GitHub Hugging Face Research paper Implementation Walk-through: Segmentation Tracking with DINOv3 One of the most practical ways to use DINOv3 is in segmentation tracking, i.e., following objects frame by frame in video streams. This is particularly useful in robotics, environmental monitoring, and medical imaging. Annotate Training Data with Encord Start by creating your video annotation dataset using Encord’s video-native platform. The Label Editor supports frame-level bitmask annotations, object tracking, interpolation, and timeline navigation making annotation fast, consistent, and precise. Load a Pretrained Model Start with pretrained DINOv3 weights, which already capture strong visual representations. import torch model = torch.hub.load('facebookresearch/dinov3', 'dinov3_vitl') model.eval() Preprocess Input Frames Prepare video frames with the same normalization pipeline used in training. from torchvision import transforms transform = transforms.Compose([ transforms.Resize(224), transforms.CenterCrop(224), transforms.ToTensor(), transforms.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)), ]) Extract Dense Patch Frames Request dense outputs to get per-patch embeddings, not just a global vector. with torch.no_grad(): features = model(input_tensor, return_dense=True) # shape: (patches × dim) Propagate Segmentation Masks Over Time Define segmentation labels on one frame, then use feature-wise similarity to propagate and refine them across later frames. This works even when objects move or lighting changes. Visualize and Evaluate Overlay the propagated masks onto video frames to assess tracking coherence and segmentation quality. Here is the Jupyter notebook by Meta AI for video segmentation tracking using a non-parametric method: run in Colab. DINOv3 Limitations While DINOv3 pushes self-supervised vision forward, it can still faces few challenges: Domain Sensitivity: DINOv3 is trained primarily on natural images e.g., ImageNet-scale data. Performance may drop in specialized domains like medical imaging, remote sensing, or industrial inspection without domain adaptation. Annotation Propagation Drift: In segmentation tracking tasks, mask propagation can accumulate errors over long sequences since it relies on feature similarity rather than explicit temporal modeling. DINO v3: Key Takeaway DINOv3 scales vision transformers to billions of parameters and diverse image sizes without supervision. Strong performance across tasks: segmentation, tracking, retrieval, and more with minimal task-specific tuning. Accessible resources: pretrained models, code, and tutorials available via Meta’s GitHub.

Curious about what’s new in OpenAI’s GPT-5? In this technical breakdown, we cover its architecture, performance benchmarks, use cases, and how it compares to OpenAI’s open-source model GPT-OSS.  What’s New in GPT-5? GPT‑5 is OpenAI’s most capable model yet. It is smarter, faster, and more useful for real-world workflows. It’s not just about scale. GPT‑5 offers high-fidelity coding, front-end UI generation, and precise debugging with just one prompt.  Example of using GPT-5 to build a data visualization playground. Source It supports massive context windows with up to 400,000 tokens via the API (272k input + 128k output). Its reasoning variant sharpens logical thinking and produces smoother, more coherent responses. For developers, new controls like ‘verbosity’ and ‘reasoning_effort’ let you customize response detail and compute use per call. Here are some of the key features: Multi-Stage Model Routing GPT-5 uses a hierarchical routing system with at least two internal models: Fast Model: Handles standard queries with low latency. Reasoning Model: Activated automatically for complex prompts or manually via phrases like “take your time” or “think step by step.” This system enables dynamic allocation of compute, reducing latency while preserving output quality.  Improved Tool Use and Function Calling GPT-5 improves tool-use capabilities: More accurate function signature interpretation Improved argument formatting and type inference Better multi-function execution in a single pass The model is also better at generating valid JSON and structured outputs, improving integration with APIs and downstream applications. Enhanced Agentic Behavior GPT-5 performs better on multi-step tasks, long-context workflows, and goal-directed reasoning. It tracks intermediate steps more reliably and reduces the need for human intervention during task planning or execution. Higher Accuracy and Safety Compared to GPT-4: Fewer hallucinations in factual and technical tasks Reduced instruction-following failures Better behavior alignment in safety-critical applications (e.g. healthcare, legal) Developer-Oriented Features GPT-5 in the API includes: Reproducibility via seed setting Improved JSON mode for structured outputs Enhanced function calling for toolchain integration  Read Introducing GPT‑5 for developers blog which is directed towards the AI and ML engineers. GPT-5 Family GPT-5 (Base) Flagship model hosted by OpenAI. Handles long-context, multimodal tasks with top-tier performance. Best Use Case: Complex tasks, agents, RAG, multimodal reasoning GPT-5 Mini Smaller, faster variant with a balance between speed and capability. Ideal for real-time workflows. Best Use Case: Lightweight agents, fast API calls, summaries GPT-5 Nano Edge-optimized version for on-device use. Reduced capabilities, but privacy-preserving and low-latency. Best Use Case: Mobile apps, embedded systems, offline agents GPT-5 pro Advanced variant built for the most challenging reasoning tasks. Uses scaled, efficient parallel test-time compute to deliver the most comprehensive answers. Best Use Case: High-stakes reasoning in science, math, health, and code. Preferred in 67.8% of expert evaluations over GPT-5 Thinking. GPT-5: Performance While OpenAI has not released detailed architecture specs or training data sources, benchmark results confirm that GPT-5 is their most capable model to date. Academic and Reasoning Benchmarks MATH(AIME 2025, no tools): GPT-5 achieves 94.6% accuracy, up from GPT-4o’s 42.1%. SWE-bench Verified: 52.8% accuracy, showing stronger coding skills without thinking mode. Healthcare (HealthBench Hard): Scores 67.2%, with thinking mode, a notable gain in domain-specific reasoning.. Multimodal Understanding (MMMU): 84.2%, performs well on tasks involving images, video, space understanding, and scientific problem-solving. Source Fine-Tuned Assistants When evaluated in agent-like assistant benchmarks e.g., coding assistants, research agents, GPT-5 demonstrates improved memory consistency, goal tracking, and function usage. It more reliably: Calls external functions using correct schemas. Maintains context across multi-turn interactions. Produces valid structured output on request. Source Reliability in Tool Use Function-calling is more robust in GPT-5. It generates tool-structured outputs with: Higher accuracy and lower hallucination rates. Fewer schema violations in JSON outputs. More stable behavior when calling multiple tools in sequence. How to Access GPT-5 GPT-5 is available through: ChatGPT (chat.openai.com): Enabled by default for pro users under the GPT-4 selector. Automatically routes to GPT-5 in most cases. OpenAI API (platform.openai.com): Accessible via the gpt-5 model family. Supports both single-call and streaming interfaces. Azure OpenAI Service: Available under the GPT-5 deployment names depending on your region and subscription. Third-party apps & integrations: GPT-5 powers assistants in Microsoft products (e.g. Word, Excel) and other OpenAI API partners. You don’t need to tweak your prompts GPT-5 works with those built for GPT-4 Turbo, but gives you better reasoning, stronger multilingual support, and longer context handling.  For more information on GPT-5, read OpenAI’s blog post. GPT-OSS: OpenAI’s Open-Weight Models OpenAI recently also released two open-weight large language models for the first time. gpt-oss-120b: ~117B total parameters, ~5.1B active per token. gpt-oss-20b: ~21B total, ~3.6B active. Source Both models use a Mixture-of-Experts (MoE) architecture. Only a subset of parameters is active during inference, which improves efficiency and reduces computational cost. Key Features Apache 2.0 license: Fully open for commercial and research use Supports 128K context: Thanks to RoPE extension and sliding window attention Compatible with open inference engines: Tested with vLLM, TGI, Hugging Face Transformers Can run on a single 24GB consumer GPU (gpt-oss-22b), or a single H100 (gpt-oss-104b) Instruction-tuned versions available: Released alongside base models  For more information, check out their blog Introducing gpt-oss. Architecture The models use Group Query Attention (GQA) and sliding window attention. These techniques help support long-context inference and improve efficiency across hardware setups. The models are trained with RoPE embeddings extended to 128K context length, making them suitable for use in RAG systems.  For more information, read its Model Card. Performance gpt-oss-120b performs competitively with OpenAI’s o4-mini: Strong results on MMLU, GPQA, AIME, and Codeforces Outperforms o3-mini in math, health, and science at a smaller scale This makes GPT‑OSS viable for production environments where hosted solutions aren’t an option. Intended Application GPT-OSS is designed to support: Long-context applications RAG Tool use and agentic workflows Instruction-following tasks How To Access GPT-OSS Download & Self‑Host (Open-Weight): The model weights (gpt-oss-120b, gpt-oss-20b) are freely downloadable under the Apache 2.0 license from Hugging Face or GitHub. Use via Inference Providers (Managed Hosting): If you prefer not to self-host, GPT‑OSS models are accessible via managed platforms like Hugging Face Inference Providers. You cannot access it via OpenAI’s hosted API or ChatGPT platforms  Implement GPT-OSS on your own by following OpenAI’s guide. Use Cases: When to Use What GPT-5 Use GPT-5 when you need top-tier performance: Handles text, image, audio, and video Strong at multilingual, spatial, and scientific reasoning Ideal for production, large context (up to 128K), and commercial deployment via API or ChatGPT GPT-OSS Use GPT-OSS when you need full control: Runs locally, inspectable weights Good for fine-tuning, domain adaptation, or academic work Ideal for building open-source tools or constrained deployments Bottom line Need accuracy and scale? Use GPT-5. Need transparency and control? Use GPT-OSS. Key Takeaways GPT‑5 is OpenAI’s most advanced model, with better reasoning, 400K context, and improved tool use. Includes Mini, Nano, and Pro variants, optimized for different use cases—from edge devices to high-stakes reasoning. GPT‑OSS offers open-weight models (120B & 20B) with MoE and 128K context, great for transparency and local use. Available via ChatGPT, API, Azure, and Microsoft apps, and works with existing GPT-4 prompts. Use GPT‑5 for performance, GPT‑OSS for control and openness.

We are thrilled to announce that Encord, the universal AI data layer for enterprise, is now available on the Google Cloud Marketplace. This launch enables direct access to our data preparation platform, helping Google Cloud users accelerate AI development cycles and improve model performance. With our availability on Google Cloud Marketplace, you can connect seamlessly to Google Cloud storage and services. Encord’s specialized data preparation capabilities deliver AI-ready data directly into your machine learning pipelines, enabling you to efficiently train and fine-tune models within your existing Google Cloud environment.  Data is the New Compute While Google Cloud delivers robust infrastructure and machine learning services, optimizing the high-quality training and evaluation data essential for successful AI deployment often remains a bottleneck, particularly for complex computer vision and multimodal tasks. Encord solves this by providing precise, scalable, and secure data labeling and management. “Bringing Encord to Google Cloud Marketplace will help customers quickly deploy, manage, and grow the data preparation solution on Google Cloud's trusted, global infrastructure," said Dai Vu, Managing Director, Marketplace & ISV GTM Programs at Google Cloud. “Encord can now securely scale and support customers on their digital transformation journeys.” Why This Matters Encord is purpose-built to help you deploy AI models faster by ensuring your raw data is transformed into AI-ready datasets. Our platform streamlines the entire ML lifecycle, from data annotation and curation to model evaluation, all within a unified environment designed for peak performance on Google Cloud. This leads to faster time to market and improved model performance. Multimodal Data Processing: Handle diverse data types including video, image, audio, and sensor fusion with specialized annotation tools. Our multimodal capabilities support complex AI applications requiring multiple input streams. Robotics & Autonomous Systems Support: Build reliable training datasets for Physical AI applications. Our platform handles LiDAR and 3D point cloud annotation with precision, crucial for navigation, object recognition, and manipulation tasks in complex real-world environments. Medical Imaging Workflow: Implement specialized annotation for DICOM, radiology, MRI and CT scans with pixel-level accuracy. Our platform maintains compliance with HIPAA, SOC 2, and GDPR while enabling the detailed labeling required for diagnostics, treatment planning, and drug discovery Empower Generative AI Workflows: Improve your Generative AI models through efficient Human-in-the-Loop (HITL) and Reinforcement Learning from Human Feedback (RLHF) workflows, ensuring your models are refined with high-quality, human-validated data. Get Started Today Encord is trusted by over 200 of the world's top AI teams at leading enterprises and research labs. With Encord now on Google Cloud Marketplace, users can easily leverage the universal AI data layer solution  to streamline your AI development, enhance data quality, and accelerate your path to production-ready AI. See how Encord fits into your ML infrastructure stack. Sign up for a free demo today!

Generative AI is making its foothold in many industries, from healthcare to marketing, driving efficiency, boosting creativity, and creating real business impact. Organizations are integrating LLMs and other foundation models into customer-facing apps, internal tools, and high-impact workflows. But as AI systems move out of the lab and into the hands of real users, one thing becomes clear: Evaluation is no longer optional, it’s foundational. On a recent webinar hosted by Encord and Weights & Biases, industry experts Oscar Evans (Encord) and Russell Ratshin (W&B) tackled the evolving demands of AI evaluation. They explored what’s missing from legacy approaches and what it takes to build infrastructure that evolves with frontier AI, rather than struggling to catch up. Here are three key signs your AI evaluation pipeline is broken and what you can do to fix it. 1. You're Only Measuring Accuracy, Not Alignment Traditional evaluation frameworks tend to over-index on objective metrics like accuracy or BLEU scores. While these are useful in narrow contexts, they fall short in the real world where AI models need to be aligned with human goals and perform on real-world,complex tasks that have nuance.  Oscar Evans put it simply during the webinar:  “If we're looking at deploying applications that are driving business impacts and are being used by humans, then the only way to make sure that these are aligned to our purpose and that these are secure is to have humans go in and test them.” AI systems can generate perfectly fluent responses that are toxic, misleading, or factually wrong. Accuracy doesn’t catch those risks, whereas alignment does. And alignment can't be assessed in a vacuum. Fix it: Implement rubric-based evaluations to assess subjective dimensions like empathy, tone, helpfulness, and safety Incorporate human-in-the-loop feedback loops, especially when fine-tuning for use cases involving users, compliance, or public exposure Measure alignment to intent, not just correctness, particularly for open-ended tasks like summarization, search, or content generation 2. Your Evaluation Is Static While Your Model Evolves While models are constantly improving and evolving, many teams still run evaluations as one-off checks, often just before deployment, not as part of a feedback loop.  This creates a dangerous gap between what the model was evaluated to do and what it’s actually doing out in the wild. This is especially true in highly complex context or dynamic environments that require precision in edge-case deployment, such as healthcare or robotics.  “Evaluations give you visibility,” Russ from W&B noted. “They show you what’s working, what isn’t, and where to tune.” Without continuous, programmatic, and human-driven evaluation pipelines, teams are flying blind as models drift, edge cases emerge, and stakes rise. Fix it: Treat evaluation as a on par with training and deployment in your ML stack Use tools like Encord and Weights & Biases to track performance across dimensions like quality, cost, latency, and safety, not just during dev, but in production Monitor model behavior post-deployment, flag regressions, and create feedback loops that drive iteration 3. You're Lacking Human Oversight Where It Matters Most LLMs can hallucinate, embed bias, or be confidently wrong. And when they're powering products used by real people, these errors become high-risk business liabilities. Programmatic checks are fast and scalable, but they often miss what only a human can see: harmful outputs, missed context, subtle tone problems, or ethical red flags.  “There’s nothing better than getting human eyes and ears on the result set,” Russ noted. Yet many teams treat human evaluation as too slow, too subjective, or too expensive to scale. That’s a mistake. In fact, strategic human evaluation is what makes scalable automation possible. Fix it: Combine programmatic metrics with structured human feedback using rubric frameworks Build internal workflows, or use platforms like Encord, to collect, structure, and act on human input efficiently Ensure diverse evaluator representation to reduce systemic bias and increase robustness When done right, human evaluation becomes not a bottleneck, but a force multiplier for AI safety, alignment, and trust. Rethinking Evaluation as Infrastructure The key takeaway: AI evaluation isn’t just a QA step. It’s core infrastructure that not only ensure success of the models being deployed in the present but also those being developed for the future.  If you're building AI that interacts with users, powers decisions, or touches production systems, your evaluation stack should be: Integrated: built directly into your development and deployment workflows Comprehensive: covering not just accuracy but subjective and contextual signals Continuous: updating and evolving as your models, data, and users change Human-Centric: because people are the ones using, trusting, and relying on the outcomes This is the key to building future-ready AI data infrastructure. Not only will this allow high-performance AI teams to keep up with progress but also implement tooling that lets them move with it. Final Thought If your AI evaluation is broken, your product risk is hidden. And if your evaluation can’t evolve, neither can your AI. The good news? The tools and practices are here. From rubric-based scoring to human-in-the-loop systems and real-time performance tracking, teams now have the building blocks to move past ad hoc evaluation and toward truly production-ready AI.  Want to see what that looks like in action? Catch the full webinar with Encord + Weights & Biases for a deep dive into real-world evaluation workflows.

Welcome to the first edition of ‘Meet the Encord Eng Leads’, a mini series where we sit down with the team behind the code to learn more about life (and engineering) at Encord. In this edition, we’re chatting with Rad, one of Encord’s founding engineers and now squad lead for our physical AI tooling team. From real-time 3D visualisation to multi-sensor fusion and robotics infrastructure, Rad’s team is working on some of the most exciting engineering problems in the AI space. We dive into what it’s like to build cutting-edge tools, the problem his squad is solving and the future of Encord!  We are also hiring for a number of engineering roles in UK & SF! You can find them here: https://encord.com/careers/ or reach out to Kerry for more info.  So Rad, let’s kick things off! What’s your role at Encord, and what are you currently working on? Rad: I joined Encord as the founding engineer 4 years ago, and over time I’ve worked across a wide range of projects. These days, I lead a squad focused on what we call physical AI tooling — building out the platform capabilities to support robotics, autonomous vehicles, and other embodied AI systems. What kind of problems are you solving right now? Rad: We’re tackling problems at the intersection of user experience, ML infrastructure, and data tooling. Think: how do you visualize millions of data points in a way that’s actually useful? How do you build labeling workflows that feel like magic but scale like enterprise software? It’s a mix of product thinking, technical architecture, and a healthy respect for browser performance limits. That means a lot of hands-on work with 3D sensor data — think LiDAR, radar, and multi-camera setups — and fusing those inputs into coherent scene reconstructions. We’re essentially building infrastructure to enable machines to perceive and reason about the real world. It’s not just about parsing pixels anymore; it’s about helping users create high-quality datasets and training pipelines for AI systems that interact physically with their environment. It’s exciting because most models today still operate in digital-only domains — text, audio, static images. But the physical world is where AI gets really interesting (and useful). Helping move the field from sci-fi to real-world impact — whether that’s safer self-driving cars or smarter home robotics — is incredibly rewarding engineering. And why does working on this problem space excite you? Rad: It’s the frontier of AI. Everyone’s focused on large language models, but what happens when those models need to drive a car or fly a drone? Suddenly, clean data, spatial awareness, and real-time feedback loops matter — a lot. That’s our domain. It’s messy, complex, and you can’t just throw more compute at the problem. You need better tools, better data, and thoughtful engineering. That’s what we’re building. So, what originally drew you to Encord? Rad: A few things: the mission, the people, and the chance to work on some very non-trivial engineering problems. We’re enabling the future of AI — helping teams working on everything from autonomous vehicles to surgical robotics. And during the interviews, it was clear this wasn’t just a smart team — it was a kind one, too. High standards, low ego. That's rare. Surgical robotics! Wow. Could you also tell us a bit about your squad? Rad: Curious, high-trust, and delightfully nerdy. We move fast, but we’re thoughtful. Everyone’s got strong opinions, but there’s no ego — just a shared desire to build great stuff. Debugging a race condition feels like a team sport, and shipping something weirdly performant gets you Slack kudos and probably a meme. It’s a good mix of serious engineering and not taking ourselves too seriously. Sounds pretty awesome. What advice would you give to someone thinking about joining the team? Rad: Be curious, be proactive, and bring your whole self. If you love solving hard problems, collaborating with smart humans, and shipping things that matter — you’ll fit right in. Oh, and don’t be afraid to jump into a conversation or share an idea. Initiative is always welcomed here. What excites you most about the future of Encord? Rad: The size of the problem we’re solving. AI is changing fast, but data tooling hasn’t caught up — especially for teams building multimodal, physical-world systems. We’re not just filling a gap; we’re building entirely new infrastructure that will become table stakes in the next few years. It feels like we’re still early — and that’s exciting. The things we’re building now are going to shape how future AI systems get trained.  And lastly, one word to describe life at Encord? Rad: Alive. In the best way. It’s fast-paced, challenging, and full of people who genuinely care. You’re never just clocking in — you’re building something that could shape the future of AI. That’s pretty cool. You can connect with Rad here. And keep your eyes peeled for the next episode! 

In the early days of computer vision, machine learning infrastructure was relatively straightforward. You collected a dataset, labeled it, trained a model, and deployed it. The process was linear and static because the models we were building didn’t need to adapt to changing environments. However, as AI applications advance, the systems we're building are no longer just models that make predictions. They are agents that perceive, decide, act, and learn in the real world. As model performance continues to exponentially improve, the infrastructure needs to be optimized for dynamic, real-world feedback loops. For high performance AI teams building for complex use cases, such as surgical robotics or autonomous driving, this future-ready infrastructure is crucial. Without it, these teams will not be able to deliver at speed and at scale, hurting their competitive edge in the market.  Let’s unpack what this shift really means, and why you need to rethink your infrastructure now, not after your next model hits a wall in production. What Model-Centric Infrastructure Looks Like In traditional ML workflows, the model was the center. Whereas surrounding infrastructure, like data collection, annotation tools, evaluation benchmarks, was all designed to feed the training process. That stack typically looked like this: Collect a dataset (manually or from a fixed pipeline) Label it once Train a model Evaluate on a benchmark Deploy  But three things have changed: The tasks are getting harder – Models are being asked to understand context, multi-modal signals, temporal dynamics, and edge cases (ex: robotics applications) The environments are dynamic – Models are no longer just processing static inputs. They operate in real-world loops: in hospitals, warehouses, factories, and embedded applications. The cost of failure has gone up – It's not just about lower accuracy anymore. A brittle perception module in a surgical robot, or a misstep in a drone’s navigation agent, can mean real-world consequences. Why We Are Shifting from Models to Agents An agent isn’t just a model. It’s a system that: Perceives its environment (via CV, audio, sensor inputs, etc.) Decides what to do (based on learned policies or planning algorithms) Acts in the world (physical or digital) Learns from its outcomes The key here is that agents learn from outcomes.  Agents don’t live in the world of fixed datasets and static benchmarks. They live in dynamic systems. And every decision they make produces new data, new edge cases, and new sources of feedback. That means your infrastructure can’t just support training. It has to support continuous improvement, or rather a feedback loop. What Agents Demand from Infrastructure Here’s what AI agents that are operating in real world, dynamic environment demand from training infra: 1. Feedback Loops Rather than a stack with a one-way flow (data → model → prediction), agents generate continuous feedback. They need infrastructure that can ingest that feedback and use it to trigger re-training, relabeling, or re-evaluation. 2. Behavior-Driven Data Ops The next critical datapoint isn't randomly sampled, it’s based on what the agent is doing wrong. The system needs to surface failure modes and edge cases in order to automatically route them into data pipelines. 3. Contextual Annotation Workflows For agents operating in multimodal environments (e.g. surgical scenes, drone footage, or robotic arms), you need annotation systems that are aware of context. This is why a tool like Encord’s multilingual editor is helpful, allowing different views of a single object to be annotated simultaneously.  Encord HTIL workflow 4. Real-Time Evaluation & Monitoring Where the real challenge with agents and complex models lie is when they are productionized. This is where failures and edge cases often come to the surface. Therefore, AI infra must be evaluated and monitored in real-world conditions.  5. Human-in-the-Loop, Where It Matters Your human experts are expensive. Don’t waste them labeling random frames. Instead, design your workflows so that humans focus on critical decisions, edge-case adjudication, and behavior-guided corrections. How to Use Encord to Build AI Infra for CV Agents At Encord, we’re building the data layer for frontier AI teams. That means we’re not just another labeling tool, or a dataset management platform. We’re helping turn raw data, model outputs, agent behaviors, and human input into a cohesive system. Let’s take some complex computer vision use cases to illustrate these points: Closing the Feedback Loop An AI-powered surgical assistant captures post-op feedback. That feedback is routed through Encord to identify mislabeled cases or new patterns, which are automatically prioritized for re-annotation and model update. Surgical video ontology in Encord Behavior-Based Data Routing An autonomous warehouse robot team uses Encord to tag failure logs. These logs automatically trigger active learning workflows, so that the most impactful data gets labeled and reintroduced into training first. Contextual, Domain-Aware Labeling In computer vision for aerial drone surveillance, users annotate multi-frame sequences with temporal dependencies. Encord enables annotation with full temporal context and behavior tagging across frames. Agricultural drone CV application Dynamic Evaluation Metrics Instead of relying on outdated benchmarks, users evaluate models live based on how agents perform in the real environment. Why This Matters for AI/ML Leaders If you're a CTO, Head of AI, or technical founder, this shift should be on your radar for one key reason: If your infrastructure is built for yesterday’s AI, you’ll spend the next 18 months patching it. We’re seeing a growing split: Companies that invest in orchestration and feedback are accelerating. Companies still on static pipelines are drowning in tech debt and firefighting. You don’t want to retrofit orchestration after your systems are in production. You want to build it in from the start, especially as agents become the dominant paradigm across CV and multi-modal AI. The AI landscape is moving faster than most infrastructure can handle. Therefore, you need infrastructure that helps those models learn, adapt, and improve in the loop. That is where Encord comes in: Build agent-aware data pipelines Annotate and evaluate in dynamic, context-rich environments Automate feedback integration and retraining triggers Maintain human oversight where it matters most Adapt infrastructure alongside AI advancement Key Takeaways The AI systems of tomorrow won’t just predict, they’ll act, adapt, and improve. They’ll live in the real world, not in your test set. And they’ll need infrastructure that can evolve with them. If you’re leading an AI team at the frontier, now’s the time to modernize your infrastructure. Invest in feedback, automation, and behavioral intelligence. Learn how Encord powers future-ready AI infrastructure →

In a recent conversations with leaders in Ag-Tech and robotics, we have dug into a common but often under-discussed challenge in applied AI: How do you build computer vision models that can actually perform in the real world? From sun glare and dusty conditions to shaky camera mounts and wildly varying plant types, the AgTech and robotics fields present some of the harshest environments for deploying AI. But these challenges are faced across industries in which conditions can vary in both training and deploying models.  In this article, we are going to explore how you can curate edge-case data to build models that perform in the real-world.   The Challenge With Variable Environments Let’s start with the baseline problem: while  CV models can be trained and evaluated in ideal conditions with clean datasets, balanced lighting, and minimal noise, many are not being trained on the right data. In turn, as soon as those models leave the lab, things fall apart fast. To take the AgTech example, AI systems are forced to deal with: Inconsistent lighting: clouds rolling over, shadows from crop canopies, backlight during golden hour Dust, water, and vibration: from machines plowing soil or navigating uneven terrain Sensor instability: shaky footage, motion blur, camera obstructions Massive biological variation: different plant species, growth stages, weed types, soil textures, and even pest interference That’s not just a harder dataset, but rather a completely different operating context. A model that performs with 92% accuracy in synthetic tests may suddenly hit 60% when exposed to edge cases, like backlit weeds partially covered by dust, in motion, and with similar coloring to the surrounding soil. This is why robustness matters more than theoretical accuracy. In the wild, your model needs to handle variability gracefully, not just perform well in ideal conditions. The True Bottleneck: Curating and Labeling the Right Data If there’s one consistent theme across all the teams we’ve worked with in AgTech and field robotics, it’s this: while having an AI data labeling pipeline is crucial to model success, labeling more data isn’t always the answer. Labeling the right data is key to ensure that variability in the real-world is accounted for while maintaining maximum efficiency across the AI data pipeline.  Annotation Fatigue Is Real Labeling thousands of field images, with weeds, crops, shadows, and motion blur, is time-consuming and expensive. For most teams, annotation quickly becomes the bottleneck in model iteration. Even more frustrating: you may end up labeling hours of video or thousands of images that add little to no model improvement. So how do the best teams tackle this? Curate for Edge Cases, Not Volume Top-performing computer vision pipelines focus on edge-case sampling, such as: Occluded or partially visible objects (e.g., items behind obstacles, people partially out of frame) Low-light, high-glare, or overexposed conditions (e.g., poorly lit warehouses, shiny surfaces, backlit scenes) Uncommon object variations or rare classes (e.g., damaged products, rare defects, unusual medical cases) Motion blur or shaky footage (e.g., handheld cameras, moving platforms, vibration-prone environments) These are the moments that hinder model performance in the real world and improving on these has an outsized impact on real-world performance. How Encord Helps Teams Go Faster This is exactly where Encord fits in,  as the data engine for teams building robust computer vision systems across industries like AgTech, robotics, healthcare, logistics, manufacturing, and more. Encord gives you the tools to focus your effort on the data that actually improves model performance. Here’s how: Curate Smarter with Visual Search & Metadata Filters Not all data is equally valuable, especially when you're looking for edge cases. Encord lets you: Search across metadata such as lighting conditions, blur, object class, and camera source Tag and retrieve examples with visual search, identifying hard-to-label or rare cases with ease Organize your dataset dynamically based on failure patterns, geography, device, or any custom field Cluster & Surface Edge Cases with ML-Powered Embeddings Finding the “long tail” of your dataset is hard. Encord helps by: Clustering visually similar images using learned embeddings Letting you surface diverse and representative samples from across the distribution Identifying outliers and edge cases that models tend to miss or misclassify Label Faster with Automation & Integrated QA Once you’ve found the right data, annotation needs to be fast and accurate. Encord delivers: Multimodal annotation tools for image, video, LiDAR, 3D, documents, and more SAM-powered segmentation and pre-labeling tools to accelerate pixel-perfect annotations Custom labeling workflows with built-in QA layers, reviewer roles, and audit logs Close the Loop from Model Evaluation to Data Re-Labeling With Encord, your annotation platform becomes part of the ML feedback loop. Use model predictions to flag weak spots or uncertain examples Automatically route failure cases back into labeling or review queues Measure annotation quality and track the impact of new data on model performance Instead of randomly sampling 10,000 images, teams can focus annotation on the 1,000 examples that actually move the needle.  👉 See how Encord works Architecture Trade-Offs That Matter for Deploying Models on the Edge Once you've built a strong, diverse dataset, the next big challenge comes during deployment, especially when you're running models on edge devices, not cloud servers. In many real-world scenarios, you’re deploying to: Embedded systems in autonomous machines, vehicles, drones, or mobile devices On-premise edge hardware in environments with limited power, compute, or connectivity Ruggedized environments with physical challenges like motion, vibration, dust, or poor lighting That makes model selection and architecture choices absolutely critical. How to Create a CV Data Pipeline  Successful computer vision teams don’t treat model development as linear. Instead, they understand that they need a continuous feedback loop.  Deploy a model in production Monitor for failure cases (missed weeds, misclassifications) Capture and curate those cases Retrain on newly labeled data Evaluate and redeploy rapidly The goal is to optimise data caution, model training and model evaluation. This in turn, will improve model performance exponentially faster. This is especially critical for teams deploying physical AI (like robotics) where safety, efficiency, and explainability are all non-negotiable. 5 Ways to Build Resilient CV Systems Based on our experience with teams across robotics, AgTech, and logistics, here are 5 principles that help CV teams succeed in unpredictable environments: 1. Design for Diversity from Day One Don’t just collect clean daytime images,  gather dusk, glare, partial occlusion, and motion blur examples upfront. A diverse dataset prevents downstream surprises. 2. Prioritize Edge Case Labeling Don’t spread your labeling budget thin across all data. Focus your annotation effort on high-impact edge cases that cause model errors. 3. Build Small, Fast, Resilient Models Your model doesn’t need to be state-of-the-art. It needs to work reliably on real hardware. Optimize for latency, size, and stability. 4. Monitor Contextually Aggregate metrics can be misleading. Monitor performance by environmental condition (e.g., lighting, terrain, sensor angle) to detect hidden weaknesses. 5. Plan for Iteration, Not Perfection You won’t get it right the first time. Build pipelines, not one-off solutions. Make retraining and annotation easy to trigger from real-world feedback. Want to Build Computer Vision That Actually Works in the Real World? If you’re working on robotics, AgTech, autonomous inspection, or any other field where computer vision needs to work in variable, high-noise environments, we’d love to hear from you. At Encord, we help teams: Curate and label edge-case data faster Build datasets optimized for robustness Evaluate and iterate models through tight feedback loops Deploy high-performing CV pipelines with compliance and scale in mind  👉 Book a demo

Generative models are being deployed across a range of use cases, from drug discovery to game design. The deployment of these models in real-world applications necessitates robust evaluation processes. However, traditional metrics can’t keep up with today’s generative AI. So we had Weights & Biases join us on a live event to explore rubric-based evaluation — a structured, multi-dimensional approach that delivers deeper insight, faster iteration, and more strategic model development. This article recaps that conversation, diving into the importance of building effective evaluation frameworks, the methodologies involved, and the future of AI evaluations.  Want a replay? Watch it here. Importance of AI Evaluations Deploying AI in production environments requires confidence in its performance. Evaluations are crucial for ensuring that AI applications deliver accurate and reliable results. They help identify and mitigate issues such as hallucinations and biases, which can affect user experience and trust. Evaluations also play a vital role in optimizing AI models across dimensions like quality, cost, latency, and safety. Traditional vs. Modern Evaluation Methods Traditional evaluation methods often rely on binary success/fail metrics or statistical comparisons against a golden source of truth. While these methods provide a baseline, they can be limited in scope, especially for applications requiring nuanced human interaction. Modern evaluation approaches incorporate rubric-based assessments, which consider subjective criteria such as friendliness, politeness, and empathy. These rubrics allow for a more comprehensive evaluation of AI models, aligning them with business and human contexts. Rubric-Based Evaluation Rubric-based evaluations offer a structured approach to assess AI models beyond traditional metrics. By defining criteria such as user experience and interaction quality, businesses can ensure their AI applications meet specific objectives. This method is customizable and can be tailored to different use cases and user groups, ensuring alignment across business operations.  Download our comprehensive rubric evaluation framework. Implementation and Iteration Implementing rubric-based evaluations involves starting with simple cases and gradually expanding to more complex scenarios. This iterative process allows for continuous improvement and optimization of AI models. By leveraging human evaluations alongside programmatic assessments, businesses can gain deeper insights into model performance and make informed decisions about deployment. Human and Programmatic Evaluations Human evaluations provide invaluable context and subjectivity that programmatic methods may lack. However, scaling human evaluations can be challenging. Programmatic evaluations, such as using large language models (LLMs) as judges, can complement human assessments by handling large datasets efficiently. Combining both approaches ensures a balanced evaluation process that mitigates biases and enhances model reliability. Key Takeaways The integration of rubric-based evaluations into AI development processes is essential for creating robust and reliable AI applications. By focusing on both human and programmatic assessments, businesses can optimize their AI models for real-world deployment, ensuring they meet the desired quality and performance standards. As AI technology continues to advance, the importance of comprehensive evaluation frameworks will only grow, driving innovation and trust in AI solutions.