In the field of image generation, OpenAI continues to push the boundaries of what’s possible. On September 20th, 2023 Sam Altman announced DALL-E 3, which is set to revolutionize the world of text-to-image generation.  Fueled by Microsoft's support, the firm is strategically harnessing ChatGPT's surging popularity to maintain its leadership in generative AI, a critical move given the escalating competition from industry titans like Google and emerging disruptors like Bard, Midjourney, and Stability AI. {{light_callout_start}} Interested in fine-tuning vision foundation models, try Encord! {{light_callout_end}} DALL-E 3: What We Know So Far DALL-E 3 is a text-to-image model which is built upon DALL-E 2 and ChatGPT. It excels in understanding and translating textual descriptions into highly detailed and accurate images.  {{light_callout_start}} Watch the demo video for DALL-E 3! {{light_callout_end}}  While this powerful AI model is still in research preview, there's already a lot to be excited about. Here's a glimpse into what we know so far about DALL-E 3: Eliminating Prompt Engineering DALL-E 3 is set to redefine how we think about generating images from text. Modern text-to-image systems often fall short by ignoring words or descriptions, thereby requiring users to master the art of prompt engineering. In contrast, DALL·E 3 represents a remarkable leap forward in our ability to generate images that precisely adhere to the text provided, eliminating the complexities of prompt engineering. Integrated seamlessly with ChatGPT, DALL·E 3 acts as a creative partner, allowing users to effortlessly bring their ideas to life by generating tailored and visually stunning images from simple sentences to detailed paragraphs. DALL-E 3 Improved Precision DALL-E 3 is set to redefine how we think about generating images from text prompts. Previously DALL-E, like other generative AI models has shown issues interpreting complex text prompts and often mixing two concepts while generating images. Unlike its predecessors, this model is designed to understand text prompts with remarkable precision, capturing nuance and detail like never before. Focus on Ethical AI OpenAI is acutely aware of the ethical considerations that come with image generation models. To address these concerns, DALL-E 3 incorporates safety measures that restrict the generation of violent, adult, or hateful content. Moreover, it has mitigations in place to avoid generating images of public figures by name, thereby safeguarding privacy and reducing the risk of misinformation. OpenAI's commitment to ethical AI is further underscored by its collaboration with red teamers and domain experts. These partnerships aim to rigorously test the model and identify and mitigate potential biases, ensuring that DALL-E 3 is a responsible and reliable tool. Just this week, OpenAI unveiled the "OpenAI Red Teaming Network," a program designed to seek out experts across diverse domains. The aim is to engage these experts in evaluating their AI models, thereby contributing to the informed assessment of risks and the implementation of mitigation strategies throughout the entire lifecycle of model and product development. Transparency  As AI-generated content becomes more prevalent, the need for transparency in identifying such content grows. OpenAI is actively researching ways to help people distinguish AI-generated images from those created by humans. They are experimenting with a provenance classifier, an internal tool designed to determine whether an image was generated by DALL-E 3. This initiative reflects OpenAI's dedication to transparency and responsible AI usage. DALL-E 3 This latest iteration of DALL-E is scheduled for an initial release in early October, starting with ChatGPT Plus and ChatGPT Enterprise customers, with subsequent availability in research labs and through its API service in the autumn. OpenAI intends to roll out DALL-E 3 in phases but has not yet confirmed a specific date for a free public release. {{light_callout_start}} When DALL-E 3 is launched, you'll discover an in-depth explanation article about it on Encord! Stay tuned! {{light_callout_end}} Recommended Topics for Pre-Release Reading To brace yourself for the release and help you dive right into it, here are some suggested topics you can explore: Transformers Transformers are foundational architectures in the field of artificial intelligence, revolutionizing the way machines process and understand sequential data. Unlike traditional models that operate sequentially, Transformers employ parallel processing, making them exceptionally efficient. They use mechanisms like attention to weigh the importance of different elements in a sequence, enabling tasks such as language translation, sentiment analysis, and image generation. Transformers have become the cornerstone of modern AI, underpinning advanced models like DALL-E, ChatGPT, etc. {{light_callout_start}} For more information about Vision Transformers read Introduction to Vision Transformers (ViT){{light_callout_end}}  Foundation Models Foundation models are the bedrock of contemporary artificial intelligence, representing a transformative breakthrough in machine learning. These models are pre-trained on vast datasets, equipping them with a broad understanding of language and knowledge. GPT-3 and DALL-E, for instance, are prominent foundation models developed by OpenAI. These models serve as versatile building blocks upon which more specialized AI systems can be constructed. After pre-training on extensive text data from the internet, they can be fine-tuned for specific tasks, including natural language understanding, text generation, and even text-to-image conversion, as seen in DALL-E 3. Their ability to generalize knowledge and adapt to diverse applications underscores their significance in AI's rapid advancement. Foundation models have become instrumental in numerous fields, including large language models, AI chatbots, content generation, and more. Their capacity to grasp context, generate coherent responses, and perform diverse language-related tasks makes them invaluable tools for developers and researchers. Moreover, the flexibility of foundation models opens doors to creative and practical applications across various industries. {{light_callout_start}} For more information about foundation models read The Full Guide to Foundation Models{{light_callout_end}}  Text-to-Image Generation Text-to-image generation is a cutting-edge field in artificial intelligence that bridges the gap between textual descriptions and visual content creation. In this remarkable domain, AI models use neural networks to translate written text into vivid, pixel-perfect images. These models understand and interpret textual input, capturing intricate details, colors, and context to produce striking visual representations. Text-to-image generation finds applications in art, design, content creation, and more, offering a powerful tool for bringing creative ideas to life. As AI in this field continues to advance, it holds the promise of revolutionizing how we communicate and create visual content, offering exciting possibilities for artists, designers, and storytellers. {{light_callout_start}} Read the paper Zero-Shot Text-to-Image Generation by A. Ramesh, et al from OpenAI to understand how DALL-E generates images! {{light_callout_end}} 

With Google gearing up to release Gemini this fall set to rival OpenAI’s GPT-Vision, it is going to be the Oppenheimer vs. Barbie of generative AI.  OpenAI and Google have been teasing their ground-breaking advancements in multimodal learning. Let's discuss what we know so far. Google’s Gemini: What We Know So Far At the May 2023 Google I/O developer conference, CEO Sundar Pichai unveiled Google's upcoming artificial intelligence (AI) system, codenamed Gemini. Developed by the esteemed DeepMind division, a collaboration between the Brain Team and DeepMind itself, Gemini represents a groundbreaking advancement in AI.  While detailed information remains confidential, recent interviews and reports have provided intriguing insights into the power and potential of Google's Gemini. {{light_callout_start}} Interested in fine-tuning foundation models, contact sales to discuss your use case. {{light_callout_end}}  Gemini’s Multimodal Integration Google CEO Sundar Pichai emphasized that Gemini combines DeepMind's AlphaGo strengths with extensive language modeling capabilities. With a multimodal design, Gemini seamlessly integrates text, images, and other data types, enabling more natural conversational abilities. Pichai also hinted at the potential for memory and planning features, which opens doors for tasks requiring advanced reasoning. Diverse Sizes and Capabilities Demis Hassabis, the CEO of DeepMind, provides insight into the versatility of Gemini. Drawing inspiration from AlphaGo's techniques such as reinforcement learning and tree search, Gemini is poised to acquire reasoning and problem-solving abilities. This "series of models" will be available in various sizes and capabilities, making it adaptable to a wide range of applications. Enhancing Accuracy and Content Quality Hassabis suggested that Gemini may employ techniques like fact-checking against sources such as Google Search and improved reinforcement learning. These measures are aimed at ensuring higher accuracy and reducing the generation of problematic or inaccurate content. Universal Personal Assistant In a recent interview, Sundar Pichai discussed Gemini's place in Google's product roadmap. He made it clear that conversational AI systems like Bard represent mere waypoints, not the ultimate goal. Pichai envisions Gemini and its future iterations as "incredible universal personal assistants," seamlessly integrated into people's daily lives, spanning various domains such as travel, work, and entertainment. He even suggests that today's chatbots will appear "trivial" compared to Gemini's capabilities within a few years. GPT-Vision: What We Know So Far OpenAI recently introduced GPT-4, a multimodal model that has the ability to process both textual and visual inputs, and in turn, generate text-based outputs. GPT-4, which was unveiled in March, was initially made available to the public through a subscription-based API with limited usage. It is speculated that the full potential of GPT-4 will be revealed in the autumn as GPT-Vision, coinciding with the launch of Google’s Gemini. GPT-4 Technical Report According to the paper published by OpenAI, the following is the current information available on GPT-Vision: Transformer-Based Architecture At its core, GPT-Vision utilizes a Transformer-based architecture that is pre-trained to predict the next token in a document, similar to its predecessors. Post-training alignment processes have further improved the model's performance, particularly in terms of factuality and adherence to desired behavior. Human-Level Performance GPT-4's capabilities are exemplified by its human-level performance on a range of professional and academic assessments. For instance, it achieves remarkable success in a simulated bar exam, with scores that rank among the top 10% of test takers. This accomplishment marks a significant improvement over its predecessor, GPT-3.5, which scored in the bottom 10% on the same test. GPT-Vision is expected to show similar performance if not better. Reliable Scaling and Infrastructure A crucial aspect of GPT-4's development involved establishing robust infrastructure and optimization methods that behave predictably across a wide range of scales. This predictability allowed us to accurately anticipate certain aspects of GPT-Vision's performance, even based on models trained with a mere fraction of the computational resources. Test-Time Techniques GPT-4 effectively leverages well-established test-time techniques developed for language models, such as few-shot prompting and chain-of-thought. These techniques enhance its adaptability and performance when handling both images and text. GPT-4 Technical Report Recommended Pre-release Reading Multimodal Learning Multimodal learning is a fascinating field within artificial intelligence that focuses on training models to understand and generate content across multiple modalities. These modalities encompass text, images, audio, and more. The main goal of multimodal learning is to empower AI systems to comprehend and generate information from various sensory inputs simultaneously. Multimodal learning demonstrates tremendous potential across numerous domains, including natural language processing, computer vision, speech recognition, and other areas where information is presented in diverse formats. {{light_callout_start}} Interested in multimodal learning? Read Introduction to Multimodal Deep Learning {{light_callout_end}}  Generative AI Generative AI refers to the development of algorithms and models that have the capacity to generate new content, such as text, images, music, or even video, based on patterns and data they've learned during training. These models are not only fascinating but also incredibly powerful, as they have the ability to create content that closely resembles human-produced work. Generative AI encompasses a range of techniques, including generative adversarial networks (GANs), autoencoders, and transformer-based models. It has wide-ranging applications, from creative content generation to data augmentation and synthesis. Transformers Transformers are a class of neural network architectures that have significantly reshaped the field of deep learning. Introduced in the landmark paper "Attention Is All You Need" by Vaswani et al. in 2017, Transformers excel at processing sequential data. They employ self-attention mechanisms to capture relationships and dependencies between elements in a sequence, making them highly adaptable for various tasks. Transformers have revolutionized natural language processing, enabling state-of-the-art performance in tasks like machine translation and text generation. Their versatility extends to other domains, including computer vision, audio processing, and reinforcement learning, making them a cornerstone in modern AI research. {{light_callout_start}} Interested in Vision Transformers? Read Introduction to Vision Transformers (ViT){{light_callout_end}}  Future Advancements in Multimodal Learning Recent Advances and Trends in Multimodal Deep Learning: A Review  Multimodal Image Description Enhanced language generation models for accurate and grammatically correct captions. Advanced attention-based image captioning mechanisms. Incorporation of external knowledge for context-aware image descriptions. Multimodal models for auto video subtitling. Multimodal Video Description Advancements in video dialogue systems for human-like interactions with AI. Exploration of audio feature extraction to improve video description in the absence of visual cues. Leveraging real-world event data for more accurate video descriptions. Research on combining video description with machine translation for efficient subtitling. Focus on making video subtitling processes cost-effective. Multimodal Visual Question Answering (VQA) Design of goal-oriented datasets to support real-time applications and specific use cases. Exploration of evaluation methods for open-ended VQA frameworks. Integration of context or linguistic information to enhance VQA performance. Adoption of context-aware image feature extraction techniques. Multimodal Speech Synthesis Enhancement of data efficiency for training End-to-End (E2E) DLTTS (Deep Learning Text-to-Speech) models. Utilization of specific context or linguistic information to bridge the gap between text and speech synthesis. Implementation of parallelization techniques to improve efficiency in DLTTS models. Integration of unpaired text and speech recordings for data-efficient training. Exploration of new feature learning techniques to address the "curse of dimensionality" in DLTTS. Research on the application of speech synthesis for voice conversion, translation, and cross-lingual speech conversion. Multimodal Emotion Recognition Development of advanced modeling and recognition techniques for non-invasive emotion analysis. Expansion of multimodal emotion recognition datasets for better representation. Investigation into the preprocessing of complex physiological signals for emotion detection. Research on the application of automated emotion recognition in real-world scenarios. Multimodal Event Detection Advancements in feature learning techniques to address the "curse of dimensionality" issue. Integration of textual data with audio and video media for comprehensive event detection. Synthesizing information from multiple social platforms using transfer learning strategies. Development of event detection models that consider real-time applications and user interactions. Designing goal-oriented datasets for event detection in specific domains and applications. Exploration of new evaluation methods for open-ended event detection frameworks. {{Training_data_CTA}}

Have you ever considered the sources from which AI models acquire language? Or the extensive effort required to curate high-quality data to power today's sophisticated language systems?  By the end of the guide, you will be able to answer the following questions: What is text annotation? What are some types of text annotation?  How is text annotation? What is text annotation? Traditionally, text annotation involves adding comments, notes, or footnotes to a body of text. This practice is commonly seen when editors review a  draft, adding  notes or useful comments (i.e. annotations) before passing it on for corrections. In the context of machine learning, the term takes on a slightly different meaning. It refers to the systematic process of labeling pieces of text to generate a ground-truth. The labeled data ensures that a supervised machine learning algorithm can accurately interpret and understand the data. What does it mean to annotate text? In the data science world, annotating text is a process that requires a deep  understanding of both the problem at hand and the data itself to identify relevant features and label them so. This can be likened to the task of labeling cats and dogs in several images for image classification.  In text classification, annotating text would mean looking at sentences and marking them, putting each in predefined categories; like labeling online reviews as positive or negative, or news clippings as fake or real. More tasks, such as labeling parts of speech (like nouns, verbs, subjects, etc.), labeling key phrases or words in a text for named entity recognition (ner) or to summarize a long article or research paper in a few hundred words all come under annotating text. A Comprehensive Guide to Named Entity Recognition (NER) (Turing.com)  What are the benefits of text annotation? Doing what we described above enables a machine learning algorithm to identify different categories and use the data corresponding to these labels to learn what the data from each category typically looks like. This speeds up the learning task and improves the algorithm’s performance in the real world. Learning without labels, while common today in NLP, is challenging as it is left to the algorithm to identify the nuances of the English language without any additional help and also recognize them when the model is put out in the real world. In text classification, for instance, a negative piece of text might be veiled in sarcasm—something that a human reader would instantly recognize, but an algorithm might just see the sarcastically positive words as just positive! Text annotations and labels are invaluable in these cases. Large companies that are developing powerful language models today also, on the other hand, rely on text annotation for a number of important use cases. For social media companies, that includes flagging inappropriate comments or posts, online forums to flag bots and spammy content, or news websites to remove fake or low-quality pieces. Even apps for basic search engines and chatbots can be trained to extract information from their queries. Image by Author What are some types of annotation styles? Since there are several tasks of varying nature for language interpretation in natural language processing, annotating and preparing the training data for each of them has a different objective. However, there are some standard approaches that cover the basic NLP tasks like classifying text and parts of text. While these may not cover generative text tasks like text summarization, they are important in understanding the different approaches to label a text. Text Classification Just as it sounds, a text classification model is meant to take a piece of text (sentence, phrase or paragraph) and determine what category it belongs to. Document classification involves the categorization of long texts, often with multiple pages. This annotation process involves the annotators reading every text sample and determining which one of the context-dependent predefined categories each sample belongs to. Typical examples are binning news clippings into various topics, sorting documents based on their contents, or as simple as looking at movie plot summaries and mapping them to a genre (as shown in some examples below). Genre Classification Dataset IMDb | Kaggle Sentiment Annotation Similar to text classification in process and strategy, the annotator plays a larger role in labeling a dataset for sentiment-related tasks. This task requires the annotator to interpret the text and look for the emotion and implicit context behind it—something that is not readily apparent to humans or machines when looking at the text. Typical examples include sentiment analysis of a subject from social media data, analyzing customer feedback or product reviews, or gauging the shift in public opinion over a period of time by tracking historical texts. Entity Annotation Often understanding natural language extends to recalling or extracting important information from a given text, such as names, various numbers, topics of interest, etc. Annotating such information (in the form of words or phrases) is called entity annotation. Annotators look for terms in a text of interest and classify them into predefined categories such as dates, countries, topics, names, addresses, zip codes, etc. A user can look up or extract only the pertinent information from large documents by using models trained on such a dataset to quickly label portions of the text. Semantic annotation involves a similar process, but the tags are often concepts and topics. Keyphrase tagging (looking for topic-dependent keywords), NER (or named entity recognition) (covering a more extensive set of entities), and parts of speech tagging (understanding grammatical structure) come under entity annotation. Intent Annotation Another approach to annotating text is to direct the interpretation of a sentence towards an action. Typically used for chatbots, intent annotation helps create datasets that can train machine learning models to determine what the writer of the text wants. In the context of a virtual assistant, a message might be a greeting, an inquiry for information, or an actionable request. A model trained on a dataset where the text is labeled using intent annotation can classify each incoming message into a fixed category and simplify the conversation ahead. Linguistic Annotation This kind of text annotation focuses on how humans engage with the language—in pronunciation, phonetic sound, parts of speech, word meanings, and structure. Some of these are important in building a text-to-speech converter that creates human-sounding voices with different accents. FLORS - Part-of-Speech Tagger How is text annotated? Now that we have established the various perspectives from which an annotator can look at their task, we can look at what a standard process of text annotation would be and how to annotate text for a machine learning problem. There is no all-encompassing playbook, but a well-defined workflow to go through the process step-by-step and a clear annotation guideline helps a ton. What are annotation guidelines? Text annotation guidelines are a set of rules and suggestions that act as a reference guide for annotators. An annotator must look at it and be able to understand the modeling objective and the purpose the labels would serve to that end. Since these guidelines dictate what is required of the final annotations, they must be set by the team familiar with the data and will use the annotations.  These guidelines can begin with one of the annotation techniques, or something customized that defines the problem and what to look for in the data. They must also define various cases, common and potentially ambiguous, the annotator might face in the data and actions to perform for each such problem.  For that purpose, they must also cover common examples found in the data and guidelines to deal with outliers, out-of-distribution samples, or other cases that might induce ambiguity while annotating. You can create an annotation workflow by beginning with a skeleton process, as shown below. Curate Annotation Guidelines Selecting a Labeling Tool Defining an Annotation Process Review and Quality Control Curate Annotation Guidelines First, define the modeling problem (classification, generation, clustering, etc.) that the team is trying to tackle with the data and the expected outcome of the annotation process like the fixed set of labels/categories, data format, and exporting instructions. This can be extended to curating the actual guidelines that are comprehensive yet easy to revisit. Selecting a labeling Tool Getting the right text annotation tools can make all the difference between a laborious and menial task and a long but efficient process. Given the prevalence of text modeling, there are several open-source labeling tools available.  undefinedundefined Below is an illustration of doccano that shows how straightforward annotating intent detection and NER is! Open Source Annotation Tool for Machine Learning Practitioners Defining an Annotation Process Once the logistics are in place, it is important to have a reproducible and error-free workflow that can accommodate multiple annotators and a uniform collection of labeled samples. Defining an annotation process includes organizing the data source and labeled data, defining the usage of the guidelines and the annotation tool, a step-by-step guide to performing the actual text annotation, the format of saving and exporting the annotations, and the review every labeled sample. Given the commonly large sizes of text data teams usually work with, ensuring a streamlined flow of incoming samples and outgoing labels and reviewing each sample (which might get challenging as one sample can be as big as a multi-page document) is essential. Review and Quality Control Along with on-the-fly review, have a collective look at the labeled data periodically to avoid generic label errors or any bias in labeling that might have come in over time. undefinedundefined It is also common to have multiple annotators label the same sample for consistency and to avoid any bias in interpretation, especially in cases where sentiment or contextual interpretation is crucial. To check for the bias and reliability of multiple human annotators, there are statistical measures that can be used to highlight undesirable trends. Comparison metrics such as Cohen’s kappa statistic measure how often two annotators agree with each other on the same set of samples, given the likelihood they would agree by chance. An example of interpreting Cohen’s kappa is shown below. Monitoring such metrics would flag disagreement and expose potential caveats in understanding the data and the problem. Understanding Interobserver Agreement: The Kappa Statistic Text Annotation: Key Takeaways This article underlines the roles text annotation plays for natural language processing use cases and details how you can get started with data annotation for text. You saw how: high-quality data can significantly impact the training process for a machine learning model. different tasks require different approaches and perspectives to annotating a text corpus; some require understanding the meaning of the text, while others require grammar and structure. guidelines and choosing the right text annotation tool can simplify large-scale data annotation and improve reliability. using strategies such as multiple annotators, quality metrics, and more can help generate high-quality labels. {{Training_data_CTA}}