OpenAI continues to demonstrate its commitment to innovation with the introduction of GPT Vision.  This exciting development expands the horizons of artificial intelligence, seamlessly integrating visual capabilities into the already impressive ChatGPT. These strides reflect OpenAI’s substantial investments in machine learning research and development, underpinned by extensive training data.  In this blog, we'll break down the GPT-4Vision system card, exploring these groundbreaking capabilities and their significance for users. {{RLHF_CTA}} GPT-4 Vision Capabilities: Visual Inputs After the exciting introduction of GPT-4 in March, there was growing anticipation for an iteration of ChatGPT that would incorporate image integration capabilities. GPT-4 has recently become accessible to the public through a subscription-based API, albeit with limited usage initially.  Recently OpenAI released GPT-4V(ision) and has equipped ChatGPT with image understanding. ChatGPT's image understanding is powered by a combination of multimodal GPT-3.5 and GPT-4 models. Leveraging their adept language reasoning skills, these models proficiently analyze a diverse range of visuals, spanning photographs, screenshots, and documents containing both text and images. {{light_callout_start}} Just days prior, OpenAI's Sam Altman unveiled DALL-E 3, an AI tool that facilitates the generation of images from text inputs, harnessing the power of ChatGPT. Read OpenAI’s DALL-E 3 Explained: Generate Images with ChatGPT for more information. {{light_callout_end}} In a recent demonstration video featuring OpenAI's co-founder Greg Brockman, the capabilities of GPT-4's vision-related functions took center stage. Over the course of this year, GPT-4V has undergone rigorous testing across a multitude of applications, consistently delivering remarkable results, yielding remarkable results.  In the following section, we share key findings from our team's comprehensive evaluations of GPT-4V in diverse computer vision tasks: Object Detection GPT4-Vision  is able to provide accurate information about objects and perform tasks like object counting, showcasing its proficiency in comprehensive image analysis and understanding. For example, in the image below, identifying humans in the image prompt is not easy. But it performs well and also identifies the problem in the detection as well. Image from Unsplash as prompt in GPT4-Vision Visual Question Answering GPT4-Vision performs well in handling follow-up questions on the image prompt. For example, when presented with a meal photograph, it adeptly identifies all the ingredients and can provide insightful suggestions or information. This underscores its capacity to elevate user experiences and deliver valuable insights. Image from Unsplash as prompt in GPT4-Vision GPT4-Vision Multiple Condition Processing It also possesses the capability to read and interpret multiple instructions simultaneously. For instance, when presented with an image containing several instructions, it can provide a coherent and informative response, showcasing its versatility in handling complex queries. Figuring out multiple parking sign rules using GPT4-Vision Data Analysis GPT-4 excels in data analysis. When confronted with a graph and tasked with providing an explanation, it goes beyond mere interpretation by offering insightful observations that significantly enhance data comprehension and analysis. Graph from GPT-4 Technical Report GPT4-Vision Deciphering Text GPT-4 is adept at deciphering handwritten notes, even when they pose a challenge for humans to read. In challenging scenarios, it maintains a high level of accuracy, with just two minor errors. Using GPT4-Vision to decipher JRR Tolkien’s letter GPT-4 Vision Capabilities: Outperforms SOTA LLMs In casual conversations, differentiating between GPT-3.5 and GPT-4 may appear subtle, but the significant contrast becomes evident when handling more intricate instructions. GPT-4 distinguishes itself as a superior choice, delivering heightened reliability and creativity, particularly when confronted with instructions of greater complexity.  To understand this difference, extensive benchmark testing was conducted, including simulations of exams originally intended for human test-takers. These benchmarks included tests like the Olympiads and AP exams, using publicly available 2022–2023 editions and without specific training for the exams. GPT-4 Technical Report The results further reveal that GPT-4 outperforms GPT-3.5, showcasing notable excellence across a spectrum of languages, including low-resource ones such as Latvian, Welsh, and Swahili. GPT-4 Technical Report OpenAI has leveraged GPT-4 to make a significant impact across multiple functions, from support and sales to content moderation and programming. Additionally, it plays a crucial role in aiding human evaluators in assessing AI outputs, marking the initiation of the second phase in OpenAI's alignment strategy GPT-4 Vision Capabilities: Enhanced Steerability OpenAI has been dedicated to enhancing different facets of their AI, with a particular focus on steerability.  In contrast to the fixed personality traits, verbosity, and style traditionally linked to ChatGPT, developers and soon-to-be ChatGPT users now have the ability to customize the AI's style and tasks to their preferences. This customization is achieved through the utilization of 'system' messages, which enable API users to personalize their AI's responses within predefined limits. This feature empowers API users to significantly personalize their AI's responses within predefined bounds. OpenAI acknowledges the continuous need for improvement, particularly in addressing the occasional challenges posed by system messages. They actively encourage users to explore and provide valuable feedback on this innovative functionality.  GPT-4 Vision: Limitation While GPT-4 demonstrates significant advancements in various aspects, it's important to recognize the limitations of its vision capabilities.  In the field of computer vision, GPT-4, much like its predecessors, encounters several challenges: Reliability Issues GPT-4 is not immune to errors when interpreting visual content. It can occasionally "hallucinate" or produce inaccurate information based on the images it analyzes. This limitation highlights the importance of exercising caution, especially in contexts where precision and accuracy are of utmost importance. Overreliance On occasion, GPT-4 may generate inaccurate information, adhere to erroneous facts, or experience lapses in task performance.  What is particularly concerning is its capacity to do so convincingly, which could potentially lead to overreliance, with users placing undue trust in its responses and risking undetected errors.  To mitigate this, OpenAI recommends a multifaceted approach, including comprehensive documentation, responsible developer communication, and promoting user scrutiny.  While GPT-4 has made strides in steerability and refined refusal behavior, it may at times provide hedged responses, inadvertently fostering a sense of overreliance. Complex Reasoning Complex reasoning involving visual elements can still be challenging for GPT-4.  It may face difficulties with nuanced, multifaceted visual tasks that demand a profound level of understanding.  For example, when tasked with solving an easy-level New York Times Sudoku puzzle, it misinterprets the puzzle question and consequently provides incorrect results. Solving NY Times puzzle-easy on GPT4-Vision Notice Row5Column3 and Row6Column3 where it should be 4 and 5 it reads it as 5 and 1. Can you find more mistakes? {{light_callout_start}} Read A Guide to Building a Sudoku Solver CV Project if you don’t want to solve the sudoku on your own! {{light_callout_end}}  GPT-4 Vision: Risk and Mitigation GPT-4, similar to its predecessors, carries inherent risks within its vision capabilities, including the potential for generating inaccurate or misleading visual information. These risks are amplified by the model's expanded capabilities.  In an effort to assess and address these potential concerns, OpenAI collaborated with over 50 experts from diverse fields to conduct rigorous testing, putting the model through its paces in high-risk areas that demand specialized knowledge. To mitigate these risks, GPT-4 employs an additional safety reward signal during Reinforcement Learning from Human Feedback (RLHF) training. This signal serves to reduce harmful outputs by teaching the model to refuse requests for unsafe or inappropriate content. The reward signal is provided by a classifier designed to judge safety boundaries and completion style based on safety-related prompts.  While these measures have substantially enhanced GPT-4's safety features compared to its predecessor, challenges persist, including the possibility of "jailbreaks" that could potentially breach usage guidelines. {{light_callout_start}} Read Guide to Reinforcement Learning from Human Feedback (RLHF) for Computer Vision for information on RLHF. {{light_callout_end}}  GPT-4 Vision: Access OpenAI Evals In its initial GPT-4 release, OpenAI emphasized its commitment to involving developers in the development process. To further this engagement, OpenAI has now open-sourced OpenAI Evals, a powerful software framework tailored for the creation and execution of benchmarks to assess models like GPT-4 at a granular level. Evals serves as a valuable tool for model development, allowing the identification of weaknesses and the prevention of performance regressions. Furthermore, it empowers users to closely monitor the evolution of various model iterations and facilitates the integration of AI capabilities into a wide array of applications. A standout feature of Evals is its adaptability, as it supports the implementation of custom evaluation logic. OpenAI has also provided predefined templates for common benchmark types, streamlining the process of creating new evaluations.  The ultimate goal is to encourage the sharing and collective development of a wide range of benchmarks, covering diverse challenges and performance aspects. ChatGPT Plus ChatGPT Plus subscribers now have access to GPT-4 on chat.openai.com, albeit with a usage cap.  OpenAI plans to adjust this cap based on demand and system performance. As traffic patterns evolve, there's the possibility of introducing a higher-volume subscription tier for GPT-4. OpenAI may also provide some level of free GPT-4 queries, enabling non-subscribers to explore and engage with this advanced AI model. API To gain API access, you are required to join the waitlist. However, for researchers focused on studying the societal impact of AI, there is an opportunity to apply for subsidized access through OpenAI's Researcher Access Program. GPT-4 Vision: Key Takeaways ChatGPT is now powered by visual capabilities making it more versatile. GPT-4 Vision can be used for various computer vision tasks like deciphering written texts, OCR, data analysis, object detection, etc. Still has limitations like hallucination similar to GPT-3.5. However, the overreliance is reduced compared to GPT-3.5 because of enhanced steerability. It’s available now to ChatGPT Plus users! {{RLHF_CTA}}

With Google gearing up to release Gemini this fall set to rival OpenAI’s GPT-Vision, it is going to be the Oppenheimer vs. Barbie of generative AI.  OpenAI and Google have been teasing their ground-breaking advancements in multimodal learning. Let's discuss what we know so far. Google’s Gemini: What We Know So Far At the May 2023 Google I/O developer conference, CEO Sundar Pichai unveiled Google's upcoming artificial intelligence (AI) system, codenamed Gemini. Developed by the esteemed DeepMind division, a collaboration between the Brain Team and DeepMind itself, Gemini represents a groundbreaking advancement in AI.  While detailed information remains confidential, recent interviews and reports have provided intriguing insights into the power and potential of Google's Gemini. {{light_callout_start}} Interested in fine-tuning foundation models, contact sales to discuss your use case. {{light_callout_end}}  Gemini’s Multimodal Integration Google CEO Sundar Pichai emphasized that Gemini combines DeepMind's AlphaGo strengths with extensive language modeling capabilities. With a multimodal design, Gemini seamlessly integrates text, images, and other data types, enabling more natural conversational abilities. Pichai also hinted at the potential for memory and planning features, which opens doors for tasks requiring advanced reasoning. Diverse Sizes and Capabilities Demis Hassabis, the CEO of DeepMind, provides insight into the versatility of Gemini. Drawing inspiration from AlphaGo's techniques such as reinforcement learning and tree search, Gemini is poised to acquire reasoning and problem-solving abilities. This "series of models" will be available in various sizes and capabilities, making it adaptable to a wide range of applications. Enhancing Accuracy and Content Quality Hassabis suggested that Gemini may employ techniques like fact-checking against sources such as Google Search and improved reinforcement learning. These measures are aimed at ensuring higher accuracy and reducing the generation of problematic or inaccurate content. Universal Personal Assistant In a recent interview, Sundar Pichai discussed Gemini's place in Google's product roadmap. He made it clear that conversational AI systems like Bard represent mere waypoints, not the ultimate goal. Pichai envisions Gemini and its future iterations as "incredible universal personal assistants," seamlessly integrated into people's daily lives, spanning various domains such as travel, work, and entertainment. He even suggests that today's chatbots will appear "trivial" compared to Gemini's capabilities within a few years. GPT-Vision: What We Know So Far OpenAI recently introduced GPT-4, a multimodal model that has the ability to process both textual and visual inputs, and in turn, generate text-based outputs. GPT-4, which was unveiled in March, was initially made available to the public through a subscription-based API with limited usage. It is speculated that the full potential of GPT-4 will be revealed in the autumn as GPT-Vision, coinciding with the launch of Google’s Gemini. GPT-4 Technical Report According to the paper published by OpenAI, the following is the current information available on GPT-Vision: Transformer-Based Architecture At its core, GPT-Vision utilizes a Transformer-based architecture that is pre-trained to predict the next token in a document, similar to its predecessors. Post-training alignment processes have further improved the model's performance, particularly in terms of factuality and adherence to desired behavior. Human-Level Performance GPT-4's capabilities are exemplified by its human-level performance on a range of professional and academic assessments. For instance, it achieves remarkable success in a simulated bar exam, with scores that rank among the top 10% of test takers. This accomplishment marks a significant improvement over its predecessor, GPT-3.5, which scored in the bottom 10% on the same test. GPT-Vision is expected to show similar performance if not better. Reliable Scaling and Infrastructure A crucial aspect of GPT-4's development involved establishing robust infrastructure and optimization methods that behave predictably across a wide range of scales. This predictability allowed us to accurately anticipate certain aspects of GPT-Vision's performance, even based on models trained with a mere fraction of the computational resources. Test-Time Techniques GPT-4 effectively leverages well-established test-time techniques developed for language models, such as few-shot prompting and chain-of-thought. These techniques enhance its adaptability and performance when handling both images and text. GPT-4 Technical Report Recommended Pre-release Reading Multimodal Learning Multimodal learning is a fascinating field within artificial intelligence that focuses on training models to understand and generate content across multiple modalities. These modalities encompass text, images, audio, and more. The main goal of multimodal learning is to empower AI systems to comprehend and generate information from various sensory inputs simultaneously. Multimodal learning demonstrates tremendous potential across numerous domains, including natural language processing, computer vision, speech recognition, and other areas where information is presented in diverse formats. {{light_callout_start}} Interested in multimodal learning? Read Introduction to Multimodal Deep Learning {{light_callout_end}}  Generative AI Generative AI refers to the development of algorithms and models that have the capacity to generate new content, such as text, images, music, or even video, based on patterns and data they've learned during training. These models are not only fascinating but also incredibly powerful, as they have the ability to create content that closely resembles human-produced work. Generative AI encompasses a range of techniques, including generative adversarial networks (GANs), autoencoders, and transformer-based models. It has wide-ranging applications, from creative content generation to data augmentation and synthesis. Transformers Transformers are a class of neural network architectures that have significantly reshaped the field of deep learning. Introduced in the landmark paper "Attention Is All You Need" by Vaswani et al. in 2017, Transformers excel at processing sequential data. They employ self-attention mechanisms to capture relationships and dependencies between elements in a sequence, making them highly adaptable for various tasks. Transformers have revolutionized natural language processing, enabling state-of-the-art performance in tasks like machine translation and text generation. Their versatility extends to other domains, including computer vision, audio processing, and reinforcement learning, making them a cornerstone in modern AI research. {{light_callout_start}} Interested in Vision Transformers? Read Introduction to Vision Transformers (ViT){{light_callout_end}}  Future Advancements in Multimodal Learning Recent Advances and Trends in Multimodal Deep Learning: A Review  Multimodal Image Description Enhanced language generation models for accurate and grammatically correct captions. Advanced attention-based image captioning mechanisms. Incorporation of external knowledge for context-aware image descriptions. Multimodal models for auto video subtitling. Multimodal Video Description Advancements in video dialogue systems for human-like interactions with AI. Exploration of audio feature extraction to improve video description in the absence of visual cues. Leveraging real-world event data for more accurate video descriptions. Research on combining video description with machine translation for efficient subtitling. Focus on making video subtitling processes cost-effective. Multimodal Visual Question Answering (VQA) Design of goal-oriented datasets to support real-time applications and specific use cases. Exploration of evaluation methods for open-ended VQA frameworks. Integration of context or linguistic information to enhance VQA performance. Adoption of context-aware image feature extraction techniques. Multimodal Speech Synthesis Enhancement of data efficiency for training End-to-End (E2E) DLTTS (Deep Learning Text-to-Speech) models. Utilization of specific context or linguistic information to bridge the gap between text and speech synthesis. Implementation of parallelization techniques to improve efficiency in DLTTS models. Integration of unpaired text and speech recordings for data-efficient training. Exploration of new feature learning techniques to address the "curse of dimensionality" in DLTTS. Research on the application of speech synthesis for voice conversion, translation, and cross-lingual speech conversion. Multimodal Emotion Recognition Development of advanced modeling and recognition techniques for non-invasive emotion analysis. Expansion of multimodal emotion recognition datasets for better representation. Investigation into the preprocessing of complex physiological signals for emotion detection. Research on the application of automated emotion recognition in real-world scenarios. Multimodal Event Detection Advancements in feature learning techniques to address the "curse of dimensionality" issue. Integration of textual data with audio and video media for comprehensive event detection. Synthesizing information from multiple social platforms using transfer learning strategies. Development of event detection models that consider real-time applications and user interactions. Designing goal-oriented datasets for event detection in specific domains and applications. Exploration of new evaluation methods for open-ended event detection frameworks. {{Training_data_CTA}}

In the field of image generation, OpenAI continues to push the boundaries of what’s possible. On September 20th, 2023 Sam Altman announced DALL-E 3, which is set to revolutionize the world of text-to-image generation.  Fueled by Microsoft's support, the firm is strategically harnessing ChatGPT's surging popularity to maintain its leadership in generative AI, a critical move given the escalating competition from industry titans like Google and emerging disruptors like Bard, Midjourney, and Stability AI. {{RLHF_CTA}} DALL-E 3: What We Know So Far DALL-E 3 is a text-to-image model which is built upon DALL-E 2 and ChatGPT. It excels in understanding and translating textual descriptions into highly detailed and accurate images.  {{light_callout_start}} Watch the demo video for DALL-E 3! {{light_callout_end}}  While this powerful AI model is still in research preview, there's already a lot to be excited about. Here's a glimpse into what we know so far about DALL-E 3: Eliminating Prompt Engineering DALL-E 3 is set to redefine how we think about generating images from text. Modern text-to-image systems often fall short by ignoring words or descriptions, thereby requiring users to master the art of prompt engineering. In contrast, DALL·E 3 represents a remarkable leap forward in our ability to generate images that precisely adhere to the text provided, eliminating the complexities of prompt engineering. Integrated seamlessly with ChatGPT, DALL·E 3 acts as a creative partner, allowing users to effortlessly bring their ideas to life by generating tailored and visually stunning images from simple sentences to detailed paragraphs. DALL-E 3 Improved Precision DALL-E 3 is set to redefine how we think about generating images from text prompts. Previously DALL-E, like other generative AI models has shown issues interpreting complex text prompts and often mixing two concepts while generating images. Unlike its predecessors, this model is designed to understand text prompts with remarkable precision, capturing nuance and detail like never before. Focus on Ethical AI OpenAI is acutely aware of the ethical considerations that come with image generation models. To address these concerns, DALL-E 3 incorporates safety measures that restrict the generation of violent, adult, or hateful content. Moreover, it has mitigations in place to avoid generating images of public figures by name, thereby safeguarding privacy and reducing the risk of misinformation. OpenAI's commitment to ethical AI is further underscored by its collaboration with red teamers and domain experts. These partnerships aim to rigorously test the model and identify and mitigate potential biases, ensuring that DALL-E 3 is a responsible and reliable tool. Just this week, OpenAI unveiled the "OpenAI Red Teaming Network," a program designed to seek out experts across diverse domains. The aim is to engage these experts in evaluating their AI models, thereby contributing to the informed assessment of risks and the implementation of mitigation strategies throughout the entire lifecycle of model and product development. Transparency  As AI-generated content becomes more prevalent, the need for transparency in identifying such content grows. OpenAI is actively researching ways to help people distinguish AI-generated images from those created by humans. They are experimenting with a provenance classifier, an internal tool designed to determine whether an image was generated by DALL-E 3. This initiative reflects OpenAI's dedication to transparency and responsible AI usage. DALL-E 3 This latest iteration of DALL-E is scheduled for an initial release in early October, starting with ChatGPT Plus and ChatGPT Enterprise customers, with subsequent availability in research labs and through its API service in the autumn. OpenAI intends to roll out DALL-E 3 in phases but has not yet confirmed a specific date for a free public release. {{light_callout_start}} When DALL-E 3 is launched, you'll discover an in-depth explanation article about it on Encord! Stay tuned! {{light_callout_end}} Recommended Topics for Pre-Release Reading To brace yourself for the release and help you dive right into it, here are some suggested topics you can explore: Transformers Transformers are foundational architectures in the field of artificial intelligence, revolutionizing the way machines process and understand sequential data. Unlike traditional models that operate sequentially, Transformers employ parallel processing, making them exceptionally efficient. They use mechanisms like attention to weigh the importance of different elements in a sequence, enabling tasks such as language translation, sentiment analysis, and image generation. Transformers have become the cornerstone of modern AI, underpinning advanced models like DALL-E, ChatGPT, etc. {{light_callout_start}} For more information about Vision Transformers read Introduction to Vision Transformers (ViT){{light_callout_end}}  Foundation Models Foundation models are the bedrock of contemporary artificial intelligence, representing a transformative breakthrough in machine learning. These models are pre-trained on vast datasets, equipping them with a broad understanding of language and knowledge. GPT-3 and DALL-E, for instance, are prominent foundation models developed by OpenAI. These models serve as versatile building blocks upon which more specialized AI systems can be constructed. After pre-training on extensive text data from the internet, they can be fine-tuned for specific tasks, including natural language understanding, text generation, and even text-to-image conversion, as seen in DALL-E 3. Their ability to generalize knowledge and adapt to diverse applications underscores their significance in AI's rapid advancement. Foundation models have become instrumental in numerous fields, including large language models, AI chatbots, content generation, and more. Their capacity to grasp context, generate coherent responses, and perform diverse language-related tasks makes them invaluable tools for developers and researchers. Moreover, the flexibility of foundation models opens doors to creative and practical applications across various industries. {{light_callout_start}} For more information about foundation models read The Full Guide to Foundation Models{{light_callout_end}}  Text-to-Image Generation Text-to-image generation is a cutting-edge field in artificial intelligence that bridges the gap between textual descriptions and visual content creation. In this remarkable domain, AI models use neural networks to translate written text into vivid, pixel-perfect images. These models understand and interpret textual input, capturing intricate details, colors, and context to produce striking visual representations. Text-to-image generation finds applications in art, design, content creation, and more, offering a powerful tool for bringing creative ideas to life. As AI in this field continues to advance, it holds the promise of revolutionizing how we communicate and create visual content, offering exciting possibilities for artists, designers, and storytellers. {{light_callout_start}} Read the paper Zero-Shot Text-to-Image Generation by A. Ramesh, et al from OpenAI to understand how DALL-E generates images! {{light_callout_end}}  {{RLHF_CTA}}