Immerse yourself in vision

Curious about what’s new in OpenAI’s GPT-5? In this technical breakdown, we cover its architecture, performance benchmarks, use cases, and how it compares to OpenAI’s open-source model GPT-OSS.  What’s New in GPT-5? GPT‑5 is OpenAI’s most capable model yet. It is smarter, faster, and more useful for real-world workflows. It’s not just about scale. GPT‑5 offers high-fidelity coding, front-end UI generation, and precise debugging with just one prompt.  Example of using GPT-5 to build a data visualization playground. Source It supports massive context windows with up to 400,000 tokens via the API (272k input + 128k output). Its reasoning variant sharpens logical thinking and produces smoother, more coherent responses. For developers, new controls like ‘verbosity’ and ‘reasoning_effort’ let you customize response detail and compute use per call. Here are some of the key features: Multi-Stage Model Routing GPT-5 uses a hierarchical routing system with at least two internal models: Fast Model: Handles standard queries with low latency. Reasoning Model: Activated automatically for complex prompts or manually via phrases like “take your time” or “think step by step.” This system enables dynamic allocation of compute, reducing latency while preserving output quality.  Improved Tool Use and Function Calling GPT-5 improves tool-use capabilities: More accurate function signature interpretation Improved argument formatting and type inference Better multi-function execution in a single pass The model is also better at generating valid JSON and structured outputs, improving integration with APIs and downstream applications. Enhanced Agentic Behavior GPT-5 performs better on multi-step tasks, long-context workflows, and goal-directed reasoning. It tracks intermediate steps more reliably and reduces the need for human intervention during task planning or execution. Higher Accuracy and Safety Compared to GPT-4: Fewer hallucinations in factual and technical tasks Reduced instruction-following failures Better behavior alignment in safety-critical applications (e.g. healthcare, legal) Developer-Oriented Features GPT-5 in the API includes: Reproducibility via seed setting Improved JSON mode for structured outputs Enhanced function calling for toolchain integration  Read Introducing GPT‑5 for developers blog which is directed towards the AI and ML engineers. GPT-5 Family GPT-5 (Base) Flagship model hosted by OpenAI. Handles long-context, multimodal tasks with top-tier performance. Best Use Case: Complex tasks, agents, RAG, multimodal reasoning GPT-5 Mini Smaller, faster variant with a balance between speed and capability. Ideal for real-time workflows. Best Use Case: Lightweight agents, fast API calls, summaries GPT-5 Nano Edge-optimized version for on-device use. Reduced capabilities, but privacy-preserving and low-latency. Best Use Case: Mobile apps, embedded systems, offline agents GPT-5 pro Advanced variant built for the most challenging reasoning tasks. Uses scaled, efficient parallel test-time compute to deliver the most comprehensive answers. Best Use Case: High-stakes reasoning in science, math, health, and code. Preferred in 67.8% of expert evaluations over GPT-5 Thinking. GPT-5: Performance While OpenAI has not released detailed architecture specs or training data sources, benchmark results confirm that GPT-5 is their most capable model to date. Academic and Reasoning Benchmarks MATH(AIME 2025, no tools): GPT-5 achieves 94.6% accuracy, up from GPT-4o’s 42.1%. SWE-bench Verified: 52.8% accuracy, showing stronger coding skills without thinking mode. Healthcare (HealthBench Hard): Scores 67.2%, with thinking mode, a notable gain in domain-specific reasoning.. Multimodal Understanding (MMMU): 84.2%, performs well on tasks involving images, video, space understanding, and scientific problem-solving. Source Fine-Tuned Assistants When evaluated in agent-like assistant benchmarks e.g., coding assistants, research agents, GPT-5 demonstrates improved memory consistency, goal tracking, and function usage. It more reliably: Calls external functions using correct schemas. Maintains context across multi-turn interactions. Produces valid structured output on request. Source Reliability in Tool Use Function-calling is more robust in GPT-5. It generates tool-structured outputs with: Higher accuracy and lower hallucination rates. Fewer schema violations in JSON outputs. More stable behavior when calling multiple tools in sequence. How to Access GPT-5 GPT-5 is available through: ChatGPT (chat.openai.com): Enabled by default for pro users under the GPT-4 selector. Automatically routes to GPT-5 in most cases. OpenAI API (platform.openai.com): Accessible via the gpt-5 model family. Supports both single-call and streaming interfaces. Azure OpenAI Service: Available under the GPT-5 deployment names depending on your region and subscription. Third-party apps & integrations: GPT-5 powers assistants in Microsoft products (e.g. Word, Excel) and other OpenAI API partners. You don’t need to tweak your prompts GPT-5 works with those built for GPT-4 Turbo, but gives you better reasoning, stronger multilingual support, and longer context handling.  For more information on GPT-5, read OpenAI’s blog post. GPT-OSS: OpenAI’s Open-Weight Models OpenAI recently also released two open-weight large language models for the first time. gpt-oss-120b: ~117B total parameters, ~5.1B active per token. gpt-oss-20b: ~21B total, ~3.6B active. Source Both models use a Mixture-of-Experts (MoE) architecture. Only a subset of parameters is active during inference, which improves efficiency and reduces computational cost. Key Features Apache 2.0 license: Fully open for commercial and research use Supports 128K context: Thanks to RoPE extension and sliding window attention Compatible with open inference engines: Tested with vLLM, TGI, Hugging Face Transformers Can run on a single 24GB consumer GPU (gpt-oss-22b), or a single H100 (gpt-oss-104b) Instruction-tuned versions available: Released alongside base models  For more information, check out their blog Introducing gpt-oss. Architecture The models use Group Query Attention (GQA) and sliding window attention. These techniques help support long-context inference and improve efficiency across hardware setups. The models are trained with RoPE embeddings extended to 128K context length, making them suitable for use in RAG systems.  For more information, read its Model Card. Performance gpt-oss-120b performs competitively with OpenAI’s o4-mini: Strong results on MMLU, GPQA, AIME, and Codeforces Outperforms o3-mini in math, health, and science at a smaller scale This makes GPT‑OSS viable for production environments where hosted solutions aren’t an option. Intended Application GPT-OSS is designed to support: Long-context applications RAG Tool use and agentic workflows Instruction-following tasks How To Access GPT-OSS Download & Self‑Host (Open-Weight): The model weights (gpt-oss-120b, gpt-oss-20b) are freely downloadable under the Apache 2.0 license from Hugging Face or GitHub. Use via Inference Providers (Managed Hosting): If you prefer not to self-host, GPT‑OSS models are accessible via managed platforms like Hugging Face Inference Providers. You cannot access it via OpenAI’s hosted API or ChatGPT platforms  Implement GPT-OSS on your own by following OpenAI’s guide. Use Cases: When to Use What GPT-5 Use GPT-5 when you need top-tier performance: Handles text, image, audio, and video Strong at multilingual, spatial, and scientific reasoning Ideal for production, large context (up to 128K), and commercial deployment via API or ChatGPT GPT-OSS Use GPT-OSS when you need full control: Runs locally, inspectable weights Good for fine-tuning, domain adaptation, or academic work Ideal for building open-source tools or constrained deployments Bottom line Need accuracy and scale? Use GPT-5. Need transparency and control? Use GPT-OSS. Key Takeaways GPT‑5 is OpenAI’s most advanced model, with better reasoning, 400K context, and improved tool use. Includes Mini, Nano, and Pro variants, optimized for different use cases—from edge devices to high-stakes reasoning. GPT‑OSS offers open-weight models (120B & 20B) with MoE and 128K context, great for transparency and local use. Available via ChatGPT, API, Azure, and Microsoft apps, and works with existing GPT-4 prompts. Use GPT‑5 for performance, GPT‑OSS for control and openness.

We are thrilled to announce that Encord, the universal AI data layer for enterprise, is now available on the Google Cloud Marketplace. This launch enables direct access to our data preparation platform, helping Google Cloud users accelerate AI development cycles and improve model performance. With our availability on Google Cloud Marketplace, you can connect seamlessly to Google Cloud storage and services. Encord’s specialized data preparation capabilities deliver AI-ready data directly into your machine learning pipelines, enabling you to efficiently train and fine-tune models within your existing Google Cloud environment.  Data is the New Compute While Google Cloud delivers robust infrastructure and machine learning services, optimizing the high-quality training and evaluation data essential for successful AI deployment often remains a bottleneck, particularly for complex computer vision and multimodal tasks. Encord solves this by providing precise, scalable, and secure data labeling and management. “Bringing Encord to Google Cloud Marketplace will help customers quickly deploy, manage, and grow the data preparation solution on Google Cloud's trusted, global infrastructure," said Dai Vu, Managing Director, Marketplace & ISV GTM Programs at Google Cloud. “Encord can now securely scale and support customers on their digital transformation journeys.” Why This Matters Encord is purpose-built to help you deploy AI models faster by ensuring your raw data is transformed into AI-ready datasets. Our platform streamlines the entire ML lifecycle, from data annotation and curation to model evaluation, all within a unified environment designed for peak performance on Google Cloud. This leads to faster time to market and improved model performance. Multimodal Data Processing: Handle diverse data types including video, image, audio, and sensor fusion with specialized annotation tools. Our multimodal capabilities support complex AI applications requiring multiple input streams. Robotics & Autonomous Systems Support: Build reliable training datasets for Physical AI applications. Our platform handles LiDAR and 3D point cloud annotation with precision, crucial for navigation, object recognition, and manipulation tasks in complex real-world environments. Medical Imaging Workflow: Implement specialized annotation for DICOM, radiology, MRI and CT scans with pixel-level accuracy. Our platform maintains compliance with HIPAA, SOC 2, and GDPR while enabling the detailed labeling required for diagnostics, treatment planning, and drug discovery Empower Generative AI Workflows: Improve your Generative AI models through efficient Human-in-the-Loop (HITL) and Reinforcement Learning from Human Feedback (RLHF) workflows, ensuring your models are refined with high-quality, human-validated data. Get Started Today Encord is trusted by over 200 of the world's top AI teams at leading enterprises and research labs. With Encord now on Google Cloud Marketplace, users can easily leverage the universal AI data layer solution  to streamline your AI development, enhance data quality, and accelerate your path to production-ready AI. See how Encord fits into your ML infrastructure stack. Sign up for a free demo today!

Generative AI is making its foothold in many industries, from healthcare to marketing, driving efficiency, boosting creativity, and creating real business impact. Organizations are integrating LLMs and other foundation models into customer-facing apps, internal tools, and high-impact workflows. But as AI systems move out of the lab and into the hands of real users, one thing becomes clear: Evaluation is no longer optional, it’s foundational. On a recent webinar hosted by Encord and Weights & Biases, industry experts Oscar Evans (Encord) and Russell Ratshin (W&B) tackled the evolving demands of AI evaluation. They explored what’s missing from legacy approaches and what it takes to build infrastructure that evolves with frontier AI, rather than struggling to catch up. Here are three key signs your AI evaluation pipeline is broken and what you can do to fix it. 1. You're Only Measuring Accuracy, Not Alignment Traditional evaluation frameworks tend to over-index on objective metrics like accuracy or BLEU scores. While these are useful in narrow contexts, they fall short in the real world where AI models need to be aligned with human goals and perform on real-world,complex tasks that have nuance.  Oscar Evans put it simply during the webinar:  “If we're looking at deploying applications that are driving business impacts and are being used by humans, then the only way to make sure that these are aligned to our purpose and that these are secure is to have humans go in and test them.” AI systems can generate perfectly fluent responses that are toxic, misleading, or factually wrong. Accuracy doesn’t catch those risks, whereas alignment does. And alignment can't be assessed in a vacuum. Fix it: Implement rubric-based evaluations to assess subjective dimensions like empathy, tone, helpfulness, and safety Incorporate human-in-the-loop feedback loops, especially when fine-tuning for use cases involving users, compliance, or public exposure Measure alignment to intent, not just correctness, particularly for open-ended tasks like summarization, search, or content generation 2. Your Evaluation Is Static While Your Model Evolves While models are constantly improving and evolving, many teams still run evaluations as one-off checks, often just before deployment, not as part of a feedback loop.  This creates a dangerous gap between what the model was evaluated to do and what it’s actually doing out in the wild. This is especially true in highly complex context or dynamic environments that require precision in edge-case deployment, such as healthcare or robotics.  “Evaluations give you visibility,” Russ from W&B noted. “They show you what’s working, what isn’t, and where to tune.” Without continuous, programmatic, and human-driven evaluation pipelines, teams are flying blind as models drift, edge cases emerge, and stakes rise. Fix it: Treat evaluation as a on par with training and deployment in your ML stack Use tools like Encord and Weights & Biases to track performance across dimensions like quality, cost, latency, and safety, not just during dev, but in production Monitor model behavior post-deployment, flag regressions, and create feedback loops that drive iteration 3. You're Lacking Human Oversight Where It Matters Most LLMs can hallucinate, embed bias, or be confidently wrong. And when they're powering products used by real people, these errors become high-risk business liabilities. Programmatic checks are fast and scalable, but they often miss what only a human can see: harmful outputs, missed context, subtle tone problems, or ethical red flags.  “There’s nothing better than getting human eyes and ears on the result set,” Russ noted. Yet many teams treat human evaluation as too slow, too subjective, or too expensive to scale. That’s a mistake. In fact, strategic human evaluation is what makes scalable automation possible. Fix it: Combine programmatic metrics with structured human feedback using rubric frameworks Build internal workflows, or use platforms like Encord, to collect, structure, and act on human input efficiently Ensure diverse evaluator representation to reduce systemic bias and increase robustness When done right, human evaluation becomes not a bottleneck, but a force multiplier for AI safety, alignment, and trust. Rethinking Evaluation as Infrastructure The key takeaway: AI evaluation isn’t just a QA step. It’s core infrastructure that not only ensure success of the models being deployed in the present but also those being developed for the future.  If you're building AI that interacts with users, powers decisions, or touches production systems, your evaluation stack should be: Integrated: built directly into your development and deployment workflows Comprehensive: covering not just accuracy but subjective and contextual signals Continuous: updating and evolving as your models, data, and users change Human-Centric: because people are the ones using, trusting, and relying on the outcomes This is the key to building future-ready AI data infrastructure. Not only will this allow high-performance AI teams to keep up with progress but also implement tooling that lets them move with it. Final Thought If your AI evaluation is broken, your product risk is hidden. And if your evaluation can’t evolve, neither can your AI. The good news? The tools and practices are here. From rubric-based scoring to human-in-the-loop systems and real-time performance tracking, teams now have the building blocks to move past ad hoc evaluation and toward truly production-ready AI.  Want to see what that looks like in action? Catch the full webinar with Encord + Weights & Biases for a deep dive into real-world evaluation workflows.

Welcome to the first edition of ‘Meet the Encord Eng Leads’, a mini series where we sit down with the team behind the code to learn more about life (and engineering) at Encord. In this edition, we’re chatting with Rad, one of Encord’s founding engineers and now squad lead for our physical AI tooling team. From real-time 3D visualisation to multi-sensor fusion and robotics infrastructure, Rad’s team is working on some of the most exciting engineering problems in the AI space. We dive into what it’s like to build cutting-edge tools, the problem his squad is solving and the future of Encord!  We are also hiring for a number of engineering roles in UK & SF! You can find them here: https://encord.com/careers/ or reach out to Kerry for more info.  So Rad, let’s kick things off! What’s your role at Encord, and what are you currently working on? Rad: I joined Encord as the founding engineer 4 years ago, and over time I’ve worked across a wide range of projects. These days, I lead a squad focused on what we call physical AI tooling — building out the platform capabilities to support robotics, autonomous vehicles, and other embodied AI systems. What kind of problems are you solving right now? Rad: We’re tackling problems at the intersection of user experience, ML infrastructure, and data tooling. Think: how do you visualize millions of data points in a way that’s actually useful? How do you build labeling workflows that feel like magic but scale like enterprise software? It’s a mix of product thinking, technical architecture, and a healthy respect for browser performance limits. That means a lot of hands-on work with 3D sensor data — think LiDAR, radar, and multi-camera setups — and fusing those inputs into coherent scene reconstructions. We’re essentially building infrastructure to enable machines to perceive and reason about the real world. It’s not just about parsing pixels anymore; it’s about helping users create high-quality datasets and training pipelines for AI systems that interact physically with their environment. It’s exciting because most models today still operate in digital-only domains — text, audio, static images. But the physical world is where AI gets really interesting (and useful). Helping move the field from sci-fi to real-world impact — whether that’s safer self-driving cars or smarter home robotics — is incredibly rewarding engineering. And why does working on this problem space excite you? Rad: It’s the frontier of AI. Everyone’s focused on large language models, but what happens when those models need to drive a car or fly a drone? Suddenly, clean data, spatial awareness, and real-time feedback loops matter — a lot. That’s our domain. It’s messy, complex, and you can’t just throw more compute at the problem. You need better tools, better data, and thoughtful engineering. That’s what we’re building. So, what originally drew you to Encord? Rad: A few things: the mission, the people, and the chance to work on some very non-trivial engineering problems. We’re enabling the future of AI — helping teams working on everything from autonomous vehicles to surgical robotics. And during the interviews, it was clear this wasn’t just a smart team — it was a kind one, too. High standards, low ego. That's rare. Surgical robotics! Wow. Could you also tell us a bit about your squad? Rad: Curious, high-trust, and delightfully nerdy. We move fast, but we’re thoughtful. Everyone’s got strong opinions, but there’s no ego — just a shared desire to build great stuff. Debugging a race condition feels like a team sport, and shipping something weirdly performant gets you Slack kudos and probably a meme. It’s a good mix of serious engineering and not taking ourselves too seriously. Sounds pretty awesome. What advice would you give to someone thinking about joining the team? Rad: Be curious, be proactive, and bring your whole self. If you love solving hard problems, collaborating with smart humans, and shipping things that matter — you’ll fit right in. Oh, and don’t be afraid to jump into a conversation or share an idea. Initiative is always welcomed here. What excites you most about the future of Encord? Rad: The size of the problem we’re solving. AI is changing fast, but data tooling hasn’t caught up — especially for teams building multimodal, physical-world systems. We’re not just filling a gap; we’re building entirely new infrastructure that will become table stakes in the next few years. It feels like we’re still early — and that’s exciting. The things we’re building now are going to shape how future AI systems get trained.  And lastly, one word to describe life at Encord? Rad: Alive. In the best way. It’s fast-paced, challenging, and full of people who genuinely care. You’re never just clocking in — you’re building something that could shape the future of AI. That’s pretty cool. You can connect with Rad here. And keep your eyes peeled for the next episode! 

In the early days of computer vision, machine learning infrastructure was relatively straightforward. You collected a dataset, labeled it, trained a model, and deployed it. The process was linear and static because the models we were building didn’t need to adapt to changing environments. However, as AI applications advance, the systems we're building are no longer just models that make predictions. They are agents that perceive, decide, act, and learn in the real world. As model performance continues to exponentially improve, the infrastructure needs to be optimized for dynamic, real-world feedback loops. For high performance AI teams building for complex use cases, such as surgical robotics or autonomous driving, this future-ready infrastructure is crucial. Without it, these teams will not be able to deliver at speed and at scale, hurting their competitive edge in the market.  Let’s unpack what this shift really means, and why you need to rethink your infrastructure now, not after your next model hits a wall in production. What Model-Centric Infrastructure Looks Like In traditional ML workflows, the model was the center. Whereas surrounding infrastructure, like data collection, annotation tools, evaluation benchmarks, was all designed to feed the training process. That stack typically looked like this: Collect a dataset (manually or from a fixed pipeline) Label it once Train a model Evaluate on a benchmark Deploy  But three things have changed: The tasks are getting harder – Models are being asked to understand context, multi-modal signals, temporal dynamics, and edge cases (ex: robotics applications) The environments are dynamic – Models are no longer just processing static inputs. They operate in real-world loops: in hospitals, warehouses, factories, and embedded applications. The cost of failure has gone up – It's not just about lower accuracy anymore. A brittle perception module in a surgical robot, or a misstep in a drone’s navigation agent, can mean real-world consequences. Why We Are Shifting from Models to Agents An agent isn’t just a model. It’s a system that: Perceives its environment (via CV, audio, sensor inputs, etc.) Decides what to do (based on learned policies or planning algorithms) Acts in the world (physical or digital) Learns from its outcomes The key here is that agents learn from outcomes.  Agents don’t live in the world of fixed datasets and static benchmarks. They live in dynamic systems. And every decision they make produces new data, new edge cases, and new sources of feedback. That means your infrastructure can’t just support training. It has to support continuous improvement, or rather a feedback loop. What Agents Demand from Infrastructure Here’s what AI agents that are operating in real world, dynamic environment demand from training infra: 1. Feedback Loops Rather than a stack with a one-way flow (data → model → prediction), agents generate continuous feedback. They need infrastructure that can ingest that feedback and use it to trigger re-training, relabeling, or re-evaluation. 2. Behavior-Driven Data Ops The next critical datapoint isn't randomly sampled, it’s based on what the agent is doing wrong. The system needs to surface failure modes and edge cases in order to automatically route them into data pipelines. 3. Contextual Annotation Workflows For agents operating in multimodal environments (e.g. surgical scenes, drone footage, or robotic arms), you need annotation systems that are aware of context. This is why a tool like Encord’s multilingual editor is helpful, allowing different views of a single object to be annotated simultaneously.  Encord HTIL workflow 4. Real-Time Evaluation & Monitoring Where the real challenge with agents and complex models lie is when they are productionized. This is where failures and edge cases often come to the surface. Therefore, AI infra must be evaluated and monitored in real-world conditions.  5. Human-in-the-Loop, Where It Matters Your human experts are expensive. Don’t waste them labeling random frames. Instead, design your workflows so that humans focus on critical decisions, edge-case adjudication, and behavior-guided corrections. How to Use Encord to Build AI Infra for CV Agents At Encord, we’re building the data layer for frontier AI teams. That means we’re not just another labeling tool, or a dataset management platform. We’re helping turn raw data, model outputs, agent behaviors, and human input into a cohesive system. Let’s take some complex computer vision use cases to illustrate these points: Closing the Feedback Loop An AI-powered surgical assistant captures post-op feedback. That feedback is routed through Encord to identify mislabeled cases or new patterns, which are automatically prioritized for re-annotation and model update. Surgical video ontology in Encord Behavior-Based Data Routing An autonomous warehouse robot team uses Encord to tag failure logs. These logs automatically trigger active learning workflows, so that the most impactful data gets labeled and reintroduced into training first. Contextual, Domain-Aware Labeling In computer vision for aerial drone surveillance, users annotate multi-frame sequences with temporal dependencies. Encord enables annotation with full temporal context and behavior tagging across frames. Agricultural drone CV application Dynamic Evaluation Metrics Instead of relying on outdated benchmarks, users evaluate models live based on how agents perform in the real environment. Why This Matters for AI/ML Leaders If you're a CTO, Head of AI, or technical founder, this shift should be on your radar for one key reason: If your infrastructure is built for yesterday’s AI, you’ll spend the next 18 months patching it. We’re seeing a growing split: Companies that invest in orchestration and feedback are accelerating. Companies still on static pipelines are drowning in tech debt and firefighting. You don’t want to retrofit orchestration after your systems are in production. You want to build it in from the start, especially as agents become the dominant paradigm across CV and multi-modal AI. The AI landscape is moving faster than most infrastructure can handle. Therefore, you need infrastructure that helps those models learn, adapt, and improve in the loop. That is where Encord comes in: Build agent-aware data pipelines Annotate and evaluate in dynamic, context-rich environments Automate feedback integration and retraining triggers Maintain human oversight where it matters most Adapt infrastructure alongside AI advancement Key Takeaways The AI systems of tomorrow won’t just predict, they’ll act, adapt, and improve. They’ll live in the real world, not in your test set. And they’ll need infrastructure that can evolve with them. If you’re leading an AI team at the frontier, now’s the time to modernize your infrastructure. Invest in feedback, automation, and behavioral intelligence. Learn how Encord powers future-ready AI infrastructure →

In a recent conversations with leaders in Ag-Tech and robotics, we have dug into a common but often under-discussed challenge in applied AI: How do you build computer vision models that can actually perform in the real world? From sun glare and dusty conditions to shaky camera mounts and wildly varying plant types, the AgTech and robotics fields present some of the harshest environments for deploying AI. But these challenges are faced across industries in which conditions can vary in both training and deploying models.  In this article, we are going to explore how you can curate edge-case data to build models that perform in the real-world.   The Challenge With Variable Environments Let’s start with the baseline problem: while  CV models can be trained and evaluated in ideal conditions with clean datasets, balanced lighting, and minimal noise, many are not being trained on the right data. In turn, as soon as those models leave the lab, things fall apart fast. To take the AgTech example, AI systems are forced to deal with: Inconsistent lighting: clouds rolling over, shadows from crop canopies, backlight during golden hour Dust, water, and vibration: from machines plowing soil or navigating uneven terrain Sensor instability: shaky footage, motion blur, camera obstructions Massive biological variation: different plant species, growth stages, weed types, soil textures, and even pest interference That’s not just a harder dataset, but rather a completely different operating context. A model that performs with 92% accuracy in synthetic tests may suddenly hit 60% when exposed to edge cases, like backlit weeds partially covered by dust, in motion, and with similar coloring to the surrounding soil. This is why robustness matters more than theoretical accuracy. In the wild, your model needs to handle variability gracefully, not just perform well in ideal conditions. The True Bottleneck: Curating and Labeling the Right Data If there’s one consistent theme across all the teams we’ve worked with in AgTech and field robotics, it’s this: while having an AI data labeling pipeline is crucial to model success, labeling more data isn’t always the answer. Labeling the right data is key to ensure that variability in the real-world is accounted for while maintaining maximum efficiency across the AI data pipeline.  Annotation Fatigue Is Real Labeling thousands of field images, with weeds, crops, shadows, and motion blur, is time-consuming and expensive. For most teams, annotation quickly becomes the bottleneck in model iteration. Even more frustrating: you may end up labeling hours of video or thousands of images that add little to no model improvement. So how do the best teams tackle this? Curate for Edge Cases, Not Volume Top-performing computer vision pipelines focus on edge-case sampling, such as: Occluded or partially visible objects (e.g., items behind obstacles, people partially out of frame) Low-light, high-glare, or overexposed conditions (e.g., poorly lit warehouses, shiny surfaces, backlit scenes) Uncommon object variations or rare classes (e.g., damaged products, rare defects, unusual medical cases) Motion blur or shaky footage (e.g., handheld cameras, moving platforms, vibration-prone environments) These are the moments that hinder model performance in the real world and improving on these has an outsized impact on real-world performance. How Encord Helps Teams Go Faster This is exactly where Encord fits in,  as the data engine for teams building robust computer vision systems across industries like AgTech, robotics, healthcare, logistics, manufacturing, and more. Encord gives you the tools to focus your effort on the data that actually improves model performance. Here’s how: Curate Smarter with Visual Search & Metadata Filters Not all data is equally valuable, especially when you're looking for edge cases. Encord lets you: Search across metadata such as lighting conditions, blur, object class, and camera source Tag and retrieve examples with visual search, identifying hard-to-label or rare cases with ease Organize your dataset dynamically based on failure patterns, geography, device, or any custom field Cluster & Surface Edge Cases with ML-Powered Embeddings Finding the “long tail” of your dataset is hard. Encord helps by: Clustering visually similar images using learned embeddings Letting you surface diverse and representative samples from across the distribution Identifying outliers and edge cases that models tend to miss or misclassify Label Faster with Automation & Integrated QA Once you’ve found the right data, annotation needs to be fast and accurate. Encord delivers: Multimodal annotation tools for image, video, LiDAR, 3D, documents, and more SAM-powered segmentation and pre-labeling tools to accelerate pixel-perfect annotations Custom labeling workflows with built-in QA layers, reviewer roles, and audit logs Close the Loop from Model Evaluation to Data Re-Labeling With Encord, your annotation platform becomes part of the ML feedback loop. Use model predictions to flag weak spots or uncertain examples Automatically route failure cases back into labeling or review queues Measure annotation quality and track the impact of new data on model performance Instead of randomly sampling 10,000 images, teams can focus annotation on the 1,000 examples that actually move the needle.  👉 See how Encord works Architecture Trade-Offs That Matter for Deploying Models on the Edge Once you've built a strong, diverse dataset, the next big challenge comes during deployment, especially when you're running models on edge devices, not cloud servers. In many real-world scenarios, you’re deploying to: Embedded systems in autonomous machines, vehicles, drones, or mobile devices On-premise edge hardware in environments with limited power, compute, or connectivity Ruggedized environments with physical challenges like motion, vibration, dust, or poor lighting That makes model selection and architecture choices absolutely critical. How to Create a CV Data Pipeline  Successful computer vision teams don’t treat model development as linear. Instead, they understand that they need a continuous feedback loop.  Deploy a model in production Monitor for failure cases (missed weeds, misclassifications) Capture and curate those cases Retrain on newly labeled data Evaluate and redeploy rapidly The goal is to optimise data caution, model training and model evaluation. This in turn, will improve model performance exponentially faster. This is especially critical for teams deploying physical AI (like robotics) where safety, efficiency, and explainability are all non-negotiable. 5 Ways to Build Resilient CV Systems Based on our experience with teams across robotics, AgTech, and logistics, here are 5 principles that help CV teams succeed in unpredictable environments: 1. Design for Diversity from Day One Don’t just collect clean daytime images,  gather dusk, glare, partial occlusion, and motion blur examples upfront. A diverse dataset prevents downstream surprises. 2. Prioritize Edge Case Labeling Don’t spread your labeling budget thin across all data. Focus your annotation effort on high-impact edge cases that cause model errors. 3. Build Small, Fast, Resilient Models Your model doesn’t need to be state-of-the-art. It needs to work reliably on real hardware. Optimize for latency, size, and stability. 4. Monitor Contextually Aggregate metrics can be misleading. Monitor performance by environmental condition (e.g., lighting, terrain, sensor angle) to detect hidden weaknesses. 5. Plan for Iteration, Not Perfection You won’t get it right the first time. Build pipelines, not one-off solutions. Make retraining and annotation easy to trigger from real-world feedback. Want to Build Computer Vision That Actually Works in the Real World? If you’re working on robotics, AgTech, autonomous inspection, or any other field where computer vision needs to work in variable, high-noise environments, we’d love to hear from you. At Encord, we help teams: Curate and label edge-case data faster Build datasets optimized for robustness Evaluate and iterate models through tight feedback loops Deploy high-performing CV pipelines with compliance and scale in mind  👉 Book a demo

Generative models are being deployed across a range of use cases, from drug discovery to game design. The deployment of these models in real-world applications necessitates robust evaluation processes. However, traditional metrics can’t keep up with today’s generative AI. So we had Weights & Biases join us on a live event to explore rubric-based evaluation — a structured, multi-dimensional approach that delivers deeper insight, faster iteration, and more strategic model development. This article recaps that conversation, diving into the importance of building effective evaluation frameworks, the methodologies involved, and the future of AI evaluations.  Want a replay? Watch it here. Importance of AI Evaluations Deploying AI in production environments requires confidence in its performance. Evaluations are crucial for ensuring that AI applications deliver accurate and reliable results. They help identify and mitigate issues such as hallucinations and biases, which can affect user experience and trust. Evaluations also play a vital role in optimizing AI models across dimensions like quality, cost, latency, and safety. Traditional vs. Modern Evaluation Methods Traditional evaluation methods often rely on binary success/fail metrics or statistical comparisons against a golden source of truth. While these methods provide a baseline, they can be limited in scope, especially for applications requiring nuanced human interaction. Modern evaluation approaches incorporate rubric-based assessments, which consider subjective criteria such as friendliness, politeness, and empathy. These rubrics allow for a more comprehensive evaluation of AI models, aligning them with business and human contexts. Rubric-Based Evaluation Rubric-based evaluations offer a structured approach to assess AI models beyond traditional metrics. By defining criteria such as user experience and interaction quality, businesses can ensure their AI applications meet specific objectives. This method is customizable and can be tailored to different use cases and user groups, ensuring alignment across business operations.  Download our comprehensive rubric evaluation framework. Implementation and Iteration Implementing rubric-based evaluations involves starting with simple cases and gradually expanding to more complex scenarios. This iterative process allows for continuous improvement and optimization of AI models. By leveraging human evaluations alongside programmatic assessments, businesses can gain deeper insights into model performance and make informed decisions about deployment. Human and Programmatic Evaluations Human evaluations provide invaluable context and subjectivity that programmatic methods may lack. However, scaling human evaluations can be challenging. Programmatic evaluations, such as using large language models (LLMs) as judges, can complement human assessments by handling large datasets efficiently. Combining both approaches ensures a balanced evaluation process that mitigates biases and enhances model reliability. Key Takeaways The integration of rubric-based evaluations into AI development processes is essential for creating robust and reliable AI applications. By focusing on both human and programmatic assessments, businesses can optimize their AI models for real-world deployment, ensuring they meet the desired quality and performance standards. As AI technology continues to advance, the importance of comprehensive evaluation frameworks will only grow, driving innovation and trust in AI solutions.

Video annotation isn’t just image annotation at scale. It’s a different challenge that traditional annotation tools cannot support.  These frame-based tools weren’t designed for the complexity of video, treating video as a sequence of disconnected images. But this has a direct impact on your model performance and ROI, leading to slow, error-prone, and costly annotation. Encord changes that. Built with native video support, Encord delivers seamless, intelligent annotation workflows that scale. Whether you're working in surgical AI, robotics, or autonomous vehicles, teams choose Encord to annotate faster and at scale to build more accurate models. Keep reading to learn why teams like SDSC, Automotus, and Standard AI rely on Encord to stay ahead. The Problem with Frame-by-Frame Annotation Frame-by-frame annotation is often the case when using open-sourced or annotation tooling that is not video native. These tools break videos down into frames, leaving each frame to be annotated as individual images. However, this has negative consequences on the AI data pipeline.  First, it can cause a fragmented workflow. Annotators must work on individual images, losing temporal continuity. Plus, object tracking becomes manual and tedious as each video yields many frames.  There is also a higher risk of inconsistency across annotations. This is because when annotating by frame, some frames may be missed due to volume and the level of detail needed. Inconsistent annotations across frames can also arise when proper context is missing. For example, bounding box drift can take place when each bounding box is drawn independently. Objects are likely to change shape and size between the start and end of a video and with manual annotation, boxes may shift, lag behind, or fluctuate.  Additionally, a lack of temporal awareness leads to quality degradation. When annotators are not able to understand the relationship between frames over time, this leads to bounding box drift, noisy labels, extra QA effort and most detrimentally, poor model performance.  Finally, it can lead to increased time spent and cost incurred. Frame-by-frame annotation is more labor-intensive for both annotators and reviewers, as each video produces thousands of frames. Instead, repeated tasks could be automated with video-native tools. What Makes Encord Different for Video Annotation However, using a video native platform, like Encord, directly mitigates these challenges. A tool that has native video support allows for keyframe interpolation, timeline navigation, and real-time preview. All of which drive greater efficiency and accuracy for developing AI at scale. Built natively for video annotation  Within Encord, video is rendered natively, allowing users to annotate and navigate across a timeline. Annotators can directly annotate on full-length videos, not broken up frames. This not only saves time but also mitigates the risk of missed frames and bounding box drift. Video annotation within the platform also supports playback with object persistence across time. And to further improve efficiency across the AI data pipeline, Encord’s real-time collaboration tools can be used on entire video sequences.  Advanced object tracking & automation Encord also offers AI-assisted tracking to automatically follow objects through frames once it has been labeled in a single or keyframe. Encord uses SAM2 to predict the place of the bounding box across subsequent frames. This supports re-identification even when objects temporarily disappear (ex: occlusion). This reduces the time spent redrawing objects in each frame and helps maintain temporal consistency.  The platform also features interpolation, model-assisted labeling, and active learning. Interpolation is a semi-automated method where the annotator marks an object at key points, and the platform fills in the labels between them by calculating smooth transitions. This leads to massive timesavings and avoids annotator fatigue, without losing accuracy.  Additionally, the active learning integration uses a feedback loop that selects frames for human annotation. Encord Active flags frames or video segments where model predictions are low-confidence. Annotators are guided to prioritize these clips. And finally, the model learns from informative samples, not redundant ones. Maintain temporal context Temporal context is critical for accurate video annotation as it relates to how objects and scenes change over time. With temporal labeling built into the UI, users can annotate events, transitions or behaviors, such a person running or car breaking.  In Encord, annotators can view and annotate frames in relation to previous and future ones. With this timeline navigation and visualisation, users can view object annotations over a video’s entire duration. This provides more context on where an object appears or changes and it is helpful for labeling intermittent objects.  Additionally, this view helps track label persistence across frames, rather than labels that are created per frame. This reduces redundant work and supports smooth object tracking, and avoids annotation drift. Finally, annotators and reviewers can play the full annotated video back to verify consistency across frames.  Why Other Platforms Fall Short Summary: Traditional platforms were designed for image annotation and retrofitted for video. Encord was purpose-built for video. The ROI of Using Encord for Video Annotation Seamless, efficient, and intelligent video annotation workflows drive both direct ROI and model accuracy. The investment in a video native annotation tool pays off through: speed, accuracy, and scalability.  Faster training & deployment  As the platform supports smart interpolation, auto-generated labels, and label persistence across a video timeline, it reduces annotation time significantly. What does this mean for ROI? Faster training data pipelines means faster model development and time to market.  By switching to Encord’s video-native platform, the Surgical Data Science Collective (SDSC) accelerated their annotation workflows by 10x while improving precision and reducing error rates from 20% to nearly zero. Encord’s seamless video rendering, Python SDK integration, and automated quality control features like object tracking and label error detection allowed SDSC to annotate complex surgical procedures at scale. Increased model accuracy Frame sync video annotation leads to better data quality through contextual, consistent annotation. This is because no frames are missed with automated video labeling. Additionally, contextual, timeline-based annotation ensures that intermittent objects are detected accurately, such as those that come in and out of the video. Plus, the more accurate the initial annotations are, the fewer QA cycles and rework. Using Encord’s intelligent visual data curation tools, Automotus was able to reduce its dataset size by 25% while increasing model performance by 20%. Encord’s platform enabled Automotus to localize objects more accurately, iterate faster, and optimize performance. Greater ability to scale production Scale is what ensures you dominate the market and competition. With the ability to annotate thousands of video files with collaborative tools, templates, and automation, scaling becomes a simple next step.  However, scaling also requires a larger, more in-sync team. Which is why support for multi-user teams, audit trails, and version control are all key features.  With Encord, Standard AI transformed its ability to scale video annotation across millions of files, cutting project kick-off times by 99.4%, accelerating video processing by 5x, and saving over $600K annually. By unifying data curation, annotation, and evaluation into a single platform with robust API and SDK support, Standard AI empowered its entire team to collaborate seamlessly and iterate rapidly, leading to production-grade retail intelligence at scale. Key Takewaways Video annotation is not easily scalable using traditional or open-source data annotation tools. The key reason is they break videos into frames, which require tedious, mistake-prone frame-by-frame annotation.  However, for deploying precise computer vision models at scale, using a video native platform is key. Encord supports keyframe interpolation, timeline navigation, and real-time preview which drive greater efficiency, accuracy, and ultimately ROI.  Encord supports smart interpolation, auto-generated labels, and label persistence, reducing annotation time significantly. Faster training data pipelines means faster model development and time to market. And with frame sync and automated video labeling, higher quality training data can be accelerated without comprising accuracy or model performance.   Make the switch to a platform that actually understands video and is built to scale. Book a demo.