How to build Semantic Visual Search with ChatGPT and CLIP

Foundation models, like Meta’s Segment Anything Model (SAM), have provided a host of benefits for data and ML teams looking to expedite the production of training data whilst improving the quality. This webinar walks you through how to go one step further and fine-tune foundation models, in particular Meta AI's SAM, to maximize relevance to your specific use case.

OpenAI’s ChatGPT and CLIP releases have revolutionised the ways in which organisations and individual contributors can ship features to their users. At Encord, we’ve focused on how the neural network (CLIP) and LLM (ChatGPT) can be combined to build an effective and powerful Semantic Visual Search. Frederik Hvilshøj, Lead ML Engineer with a PhD in Generative AI, joins Eric Landau, CEO and Co-Founder of Encord, to provide actionable insights into how to build this function from scratch.

This edition of Encord’s Fireside Chats sees Victor Prisacariu, of the University of Oxford and Niantic, sit down with Eric Landau, Encord’s CEO and Co-Founder, to discuss recent developments in AI computer vision and machine learning. Victor’s focus currently lies in real-time Augmented Reality on mobile and wearable platforms, having co-founded 6D.ai which was later acquired by Niantic in March 2020. With his wide and varied experience of the industry, Victor touched crucial areas of AR as well as discussing his work at Niantic in-depth.

With Foundational Models increasing in prominence, Encord's President and Co-Founder sat down with our Lead ML Engineer to dissect Meta's new Visual Foundation Model: Segment Anything (SAM). After combining the model with Grounding-DINO to allow for zero-shot segmentation, they compared it to a SOTA Mask-RCNN model to see whether the development of SAM really is revolutionary for segmentation.

With the proliferation of use cases for ChatGPT, we set out to investigate whether ChatGPT could be used to make improvements in other AI systems. We tested it on a practical problem in a modality of AI in which it has not been trained on - computer vision - and reported the results. ChatGPT's suggestions achieved on average a 10.1% improvement in precision and a 34.4% improvement in recall over our random sample, using a purely data-centric metric-driven approach. Eric Landau, Encord's Co-founder and CEO, sits down with the Data-Centric AI Community to share more about the process & lessons learned!

Encord's CEO & Co-founder, Eric Landau, sat down with Luc Vincent, VP of AI at Meta & also Executive Advisor at Encord - as he took us through some of the lessons learned from his career, from building Lyft's first autonomous vehicle organization, to Google's geo imagery division, and now the metaverse at Meta. Luc also touched on: the challenges and lessons learned from setting up & scaling world-class computer vision orgs that are pushing the boundaries of AI, the working environment and culture in the early days of Google, the ML applications and promising projects he's most excited about going into the next decade. And what it'll take for the winners to get ahead!

Watch previous

Watch next