Finding a reliable ecosystem to scale model development

Introducing Customer: Neurons  Neurons has been a leading company in the field of neuroscience marketing for more than 10 years. With their expertise in understanding how humans react to content, they have developed AI-powered solutions to enable companies to create predictable marketing content at scale. Their mission is to eliminate biases and guesswork from marketing strategies. As a pioneer in this domain, Neurons faced significant challenges in their data annotation process due to the sheer volume of label categories and the growing number of assets requiring annotation. We sat down with Dennis Green-Lieber (Director of Product and Engineering) and Konstantina Kaisheva (Innovation project manager) to discuss how Encord alleviated these challenges. Problem: Challenges with in-house solution Neurons, historically, have relied on manual annotation by their internal team using their in-house annotation platform. With a growing volume of assets requiring annotation, they found it increasingly challenging to scale their operations using their in-house solutions. They also encountered difficulties in ensuring precision and consistency when labeling large volumes of data, resulting in delays in getting their models to production. Solution: Scaling with Encord's end-to-end platform Seeking a solution to their data annotation challenges, Neurons turned to Encord’s platform. They recognized the need for manual and automated labeling capabilities, so they were drawn to Encord’s comprehensive offering. With Encord, Neurons found a user-friendly platform that facilitated a precise and efficient annotation process. They appreciated the ability to define labeling instructions/ontologies tailored to their specific requirements, ensuring accuracy in their data sets. Encord's platform offered features for quality assurance, workflow management, and data categorization streamlining Neurons’ annotation workflows. They were particularly impressed by Encord's flexibility, which allowed them to adapt labeling criteria for different content types. Konstantina, the R&D manager of the AI team, noted her happiness with the function of model evaluations. Recognizing its potential as an invaluable tool for the future, she highlighted its ability to provide insights into the performance and accuracy of its AI models. "It's a very easy platform to operate. What I really liked was the whole offering of Encord for both automated labeling and manual work, as precision is incredibly important to us."- Konstantina Kaisheva   Result: Efficient and fast annotation With Encord’s platform, Neurons achieved marked improvements in their data annotation process. With the appropriate workflows and quality assurance, they were able to maintain precision and consistency even when they were handling larger volumes of data. This led to building a higher-quality dataset for their AI models.  Neuron’s newly launched Copilot, provides actionable insights for marketing decision makers. This is run by AI models which require large volumes of precisely annotated datasets, which Encord enabled them to achieve. By eliminating guesswork and biases, they empowered their clients to make informed decisions and optimize their marketing strategies effectively. Overall, Encord’s platform proved to be a significant asset for Neurons, enabling them to overcome their data annotation challenges. With a newfound ability to make data-driven decisions, the company saw a tangible improvement in its ability to achieve their desired model outcomes.

CONXAI is building an AI platform for the Architecture, Engineering and Construction industries to contextualise different data and transform them into actionable insights. CONXAI, however, encountered challenges with optimizing datasets, reviewing labels, and managing large volumes of data with their in-house data annotation solution. This is where Encord came in - CONXAI was looking for an end-to-end solution for data management and curation, annotation, and evaluation. Introducing Customer: CONXAI CONXAI’s goal is to help AEC teams perform better by organizing and making sense of the vast amount of data generated during different stages of construction projects. They specialize in making data more useful, especially since a lot of project data often goes unused. Their ultimate aim is to help AEC professionals use AI effectively to improve efficiency and tackle challenges in their projects. We sat down with Markus Kittel, AI Product Development Manager at CONXAI, to discuss his work overseeing the product roadmap, and their exciting plans ahead for the business. Problem: Challenges in Data Curation and Management CONXAI's approach involves working with large unstructured datasets, which leads to challenges in effectively managing and curating project data. Their initial reliance on their in-house solution for data annotation proved to be problematic as the volume of data increased. Like many in-house tools, it was prone to frequent malfunctions, obscured the data it processed, and lacked mechanisms for reviewing annotations. Additionally, scalability was a major concern, as the in-house tool struggled to handle the increasing volume and complexity of project data. Without a reliable and scalable data management system in place, they faced difficulties in optimizing datasets and analyzing data effectively. As a result, CONXAI recognized the pressing need for a comprehensive solution that could streamline its data curation and management processes, enabling it to unlock the full potential of AI-driven insights within the AEC industry. CONXAI were also in need of a solution where data security took precedence, enabling data to remain within CONXAI servers and be accessed via an API or SDK. Solution: Encord Provides a Unified Platform for Data Curation and Management “With other labeling tools, we needed to integrate another tool for data management and exploration capabilities, but Encord combined the two needs and provided a single comprehensive solution, along with excellent customer care and support,” Markus says. To address these challenges, CONXAI explored various annotation tools. They were searching for a single platform that could handle data curation and management seamlessly. Encord's Annotate and Encord Active emerged as the ideal solution, offering a comprehensive platform to streamline CONXAI’s operations. As Markus says “We connect Encord Active with our large dataset and then use metrics to prioritize building a collection of images. This collection is then sent to Encord Annotate for labeling images in preparation for training. And all this without the data leaving our server.” Result: 60% Increase in Labeling Speed With the adoption of Encord into the data pipeline, CONXAI witnessed significant improvements in its data management processes. Encord facilitated the transformation of unstructured data into labeled, training-ready, datasets. The intuitive interface of Encord's Annotate tool simplified the annotation process for CONXAI's team, while also providing robust label review capabilities. Moreover, Encord's Active platform allowed CONXAI to efficiently curate and evaluate their datasets. “The labeling speed of the annotation team improved to almost 60% compared to when using their in-house tool.” - Markus Kittel  CONXAI was able to curate over 40k images with Encord Active. They were then able to efficiently evaluate and prioritize these images based on metrics, facilitating streamlined data management and enhanced decision-making processes within their operations. CONXAI were able to contribute to Encord’s product roadmap by identifying that mapping relationships between labels in their ontology would enhance model performance. The Encord team were able to deploy this functionality, resulting in an improved user experience for CONXAI. Overall, using Encord led to enhanced robustness, simplified data pipelines, and a remarkable 60% increase in labeling speed compared to CONXAI's previous in-house tool. This demonstrates how adopting an end-to-end platform with annotation, curation, and evaluation capabilities provides the best solution for computer vision teams. 

Introduction A new paper published in MDPI (Multidisciplinary Digital Publishing Institute) demonstrates how, using the Encord platform, researchers at Harvard Medical School, Massachusetts General Hospital, and Brigham and Women’s Hospital were able to reduce vascular ultrasound annotation time from days to minutes and run automated analyses of their datasets. Using Encord, the team was able to: Create their first segmentation models by labeling only a handful of images Cut annotation time through segmentation models by an order of magnitude Visually explore their dataset and identify problematic areas - in their case, the impact of blur on their dataset Evaluate the performance of their segmentation models in the Encord platform  Problem: DUS Image Annotation is Resource-Intensive and Prone to Human Error Medical imaging, particularly Arterial Duplex Ultrasound (DUS), plays a crucial role in diagnosing and managing vascular diseases like Popliteal Artery Aneurysms (PAAs). The traditional method of analyzing DUS images relies heavily on manual annotation by skilled medical professionals. This process is fraught with challenges: Time-consuming—especially with the growing volume of medical imaging data. Prone to human error. Heavily dependent on expertise and experience - furthering how resource-intensive the process becomes The subjective nature of manual annotations can lead to inconsistent measurements and interpretations due to inter- and intra-observer variability during annotation. This raises concerns about the reliability and reproducibility of the results and could impact the accuracy of diagnoses and treatment plans for patients. The primary issue in this research paper lies in precisely annotating the inner and outer lumens of the artery in images - a critical step for accurate measurement and subsequent treatment planning. Solution: Encord Annotate to Auto-Label DUS Images and Encord Active to Validate Model Performance The study tested the feasibility of the Encord platform to create an automated model that segments the inner and outer lumen within PAA DUS. Using image segmentation to find the largest diameter and thrombus area within PAAs helped standardize DUS measurements that are important for making decisions about surgery.  Using Encord Annotate for Automated Annotation The researchers collected and prepared (deidentification and extraction) a dataset comprising DUS images of PAAs for upload to Encord before annotating a few images to serve as ground truth for the annotation models using Encord Annotate. Using Encord Annotate’s automated labeling feature, they could generate segmentation masks for unlabeled images. This reduced the time and effort required for DUS image analysis while minimizing the potential for human error.  Using Encord Active to Select the Best-Performing Model They trained three models and validated them with Encord Active on the annotated images (20, 60, and 80 sets). Encord Active enabled the researchers to understand the performance metrics that helped them select the best model for segmenting the inner and outer lumens of the popliteal artery with high precision.  After training models on image subsets, we tested them within the Encord platform. We selected the desired tests in the analysis tab of the project, and after a runtime period, the platform presented calculations of true positives, false negatives, mAP, IoU, and blur. The report referenced Encord’s ability to seamlessly integrate into clinical processes with a user-friendly interface, simple onboarding, and rapid annotation workflows as crucial to the study's success. For healthcare practitioners who use the platform, this improves their diagnostic process without disrupting established procedures. Results Encord Reduced Annotation Time from Days to Minutes Where manual annotation could take several minutes per image, the researchers accomplished the task in a fraction of the time using Encord. Their workflow went from relying on RPVI-certified physicians manually annotating DUS images that took days to use Encord to annotate a few images, train models, and auto-label unlabeled images in minutes.  This efficiency proves crucial in clinical settings, where timely diagnosis and treatment decisions can significantly impact patient outcomes. Figure 1. AI segmentation classifications on duplex ultrasound images. (A) Outer polygon true-positive classification, where the color green indicates a correct segmentation. (B) Outer polygon false-positive classification, where red indicates an incorrect segmentation. (C) Inner polygon true-positive classification, where the color green indicates a correct segmentation. (D) Inner polygon false-positive classification, where red indicates an incorrect segmentation. Better Evaluation and Observability of Model Performance with Encord Active The researchers quantitatively assessed the performance of the three models with Encord Active providing analytics on the following metrics:  mean Average Precision (mAP).  Intersection over Union (IoU). True Positive Rate (TPR). Encord Active calculated the outer polygon's mAP to be 0.85 for the 20-image model, 0.06 for the 60-image model, and 0 for the 80-image model. The mAP of the inner polygon was 0.23 for the 20-image, 60-image, and 80-image models. The true-positive rate (TPR) for the inner polygon remained at 0.23. See the full results in the table below: “With regard to the models for outer and inner polygons, the outer polygon model outperformed the inner polygon model on every metric. The outer polygon demonstrated almost equal precision and recall at 0.85. The mAP for the outer polygon model was 0.85 with a true-positive rate of 0.86, which is comparable to other clinically used high-performing models for US segmentation.” With Encord Active automatically providing model evaluation analytics, the team instantly discovered the model's strengths and weaknesses. For every model they trained, Active provided breakdowns and graphs on its performance, including the ability to visually explore the regions the model incorrectly segmented vs. the ground truth. Encord Active Uncovered Blurry DUS Images that Could Degrade Annotation Model Performance The researchers used Encord Active to explore the model's performance depending on the blur level, allowing them to visually interact with varying levels of blur in their dataset to understand how this impacted model performance. The paper states, “Intuitively, our analysis found that as the images became blurrier, the model precision declined, and false-negative rates increased... Removing blur from—or augmenting—blur in images can be important for training accurate AI models.” Conclusion In summary, the platform’s intuitive navigation, complemented by tutorials for both model training and analysis, allowed for straightforward operationalization of the model training system among members of the research team. The results were displayed in an understandable format and interpreted within the following discussion. The findings have far-reaching consequences for medical imaging and diagnosis. The researchers greatly improved the accuracy, reliability, and efficiency of DUS image analysis by auto-annotating images with Encord Annotate and validating annotation models with Encord Active. This could result in potentially better patient care, treatment planning, and diagnostic procedures. At Encord, we are committed to continually providing healthcare practitioners and physicians with the data-centric AI platform they need to improve their medical imaging and analysis workflows.  We’re proud of the work the researchers were able to accomplish and how Encord is paving the way for broader applications of AI in various aspects of medical diagnostics.   📑 Read the full paper on MDPI (Multidisciplinary Digital Publishing Institute).