OpenAI DevDay 2024 - What No One is Talking About!

Okay, so OpenAI yesterday had their dev day and they announced pretty huge improvements to their API. Now, although there were no new models, here are the four updates that they are making. Real Time API, vision, fine tuning, prompt caching and distillation. Apart from this, they also shared a lot of details regarding their RAC system and structured JSON output. These are not covered in the blog posts, but we are going to look at those details later in the video. And I'm personally interested in RAG and structured output. But before that, let's have a quick look at these four different improvements or features. The first one is Real Time API. So now developers can build fast speech to speech experiences in their applications. Speech to speech systems were possible before this, but you would use something like Whisper to transcribe input audio. Then you will use an LLM to generate a response and there would be usually another model which will convert that text to speech. The problem with that approach is it's pretty slow. You lose a lot of emotion, emphasis and accents during these translations. But now with the same chat Completion API, the API can receive an audio input and generate audio responses, just like the advanced voice mode in ChatGPT. This is going to be pretty amazing. Not only that, just like in LLM, the Chat Completion API with the voice mode can also take actions on behalf of the user using function calling. So imagine you're having a conversation with the assistant. There is a trigger word that triggers actions. For example, the assistant could place an order on behalf of the user or retrieve relevant customer information. So there are a lot of applications that you can build, especially if you have this function calling ability built into the assistant. And with this you will be able to get both the speech as well as text output. Now the major question is how expensive this is going to be. The answer is it is going to be pretty expensive. The text input tokens are priced at $5 per million tokens and $20 per output tokens. If we look at historical data within five or six months we're going to see a substantial reduction in these prices. We have seen this across all the different models, including GPT4, O Gemini and even the Cloud series from Anthropic. Now they're saying that it's not going to be only limited to voice. So in future there could be additional modalities like vision and video. Hopefully we are going to have a very similar experience to what was shown in the original GPT4 or demos. Once I get access, I'll be creating some tutorials on the real time API. Now the second one is vision fine tuning to the API. So now you can fine tune GPT4 with images and text to improve vision capabilities. I think this is going to have some major industrial applications, especially if you're using computer vision. Now you can directly fine tune GPT4O for vision tasks. The way you do it is you will just provide a JSON where you have the information of what the image contains in terms of text, the actual image, and you create your data set and you fine tune it exactly the way you fine tune a text model within the playground. So here is an example. There's a company called Grab, which is a leading food delivery and ride sharing company. Think of this as Uber in Asia and they used the vision fine tuning capabilities of GPT4O to improve their models. Now here's another example of another company who are creating autonomous agents which can interact with websites and visual elements so it's able to identify different elements in web pages and interact with those. They are offering 1 million training tokens per day for free through October 31, 2024 to fine tune GPT4O images. The inference cost is 3.75 dollars per million input tokens and 15 dollars per million output tokens. Now keep in mind, since it's a vision model, so the number of tokens used are going to be dependent on the size of the input images. So you will have to account for that. I think there was some confusion in the beginning GPT4O was released on how these images are tokenized, but I'll highly recommend to look at the pricing page now before looking at how OpenAI recommends doing rag and structured output, let's look at the third announcement which is prompt caching in the API. This is the feature that was announced by both Google and Anthropic a while back. OpenAI is just playing catch up, but the way they have implemented prompt caching is very different than Google's and Anthropic's implementation. So they are saying that starting today prompt caching is automatically applied to the latest version of GPT4 mini, 01 preview and 01 mini, and they're going to be discounted compared to the normal prompts. Although the discount that you get here is not as aggressive as what Gemini and anthropic offering. But OpenAI's implementation is different and the way they are doing it is that the ABI calls to supported models will automatically benefit from prompt caching on prompts longer than 1024 tokens. So anything longer than 1024 tokens that is going to be automatically cached using their prompt caching feature. Now in case of both Gemini and Anthropic you have to identify the tokens but seems like they aren't doing it automatically. I am going to look at their implementation in a separate video because I think caching or prompt caching is an essential part of any LLM based application. The last one before looking at some of the RAG implementations is model distillation in the API. Now you can fine tune a cost efficient model with the outputs from a larger Frontier model on the OpenAI platform. This is a concept that both Google and Meta has explored a lot. So Google's smaller models like Flash is a distillation from bigger Pro model. Similarly the smaller LLAMA models are distillate version of the bigger models. So now they enable the distillation of smaller models something like GPT4O mini from bigger models something like 01 preview going to create a separate video because I think this one is a very interesting topic. And as a reminder you get a lot of free tokens till the end of October so make sure to take advantage of that in this section we're going to talk about how OpenAI is building better retrieval and LLM based systems and what are the different criterias and parameters that you need to consider. Most of the images and content I'm about to show you are courtesy of Greg who was at the OpenAI Dev Day and shared some of his thoughts on the different topics that were covered. Now I'm personally really interested in building retrieval system. Here are some results of the experiments that OpenAI shared. On the far left you can see a simple retrieval system with cosine similarity that's usually used in standard or naive RAC system. You can get up to 45% accuracy of retrieval which is pretty bad. But then you can explore things like hide, which is hypothetical document embedding retrieval or fine tuning the embedding models. This could potentially give you a boost, but it's probably not that great. Now they show that if you use chunking or embedding experiments so you try to optimize those two things, you can get a pretty substantial increase, that is about 50% increase. But if you add something like re ranking and classification step on top of this, this will give you another pretty huge boost. And in most of the systems that I have seen in production, re ranking is a very critical component. Now they recommend to use query expansion, prompt engineering and tool use to get better performance improvement. In my experience query expansion is a critical component because people who are asking questions, probably are not able to formulate it correctly, but you can use an LLM to rewrite those questions and as a result you're going to get better questions and hence better retrieval. The most interesting thing is tool use, which is function calling. That will add to the context of the LLM. If you want to learn more about these concepts, I cover them in a lot more details in my Advanced RAG Techniques course. Link to that course is in the video description. And for any LLM based system, or any system in general, you need to consider the accuracy target, latency target, and what are your cost targets. In order to do those you need to have evolves or test data sets because if you can't measure it, you cannot improve it. Now here's a quick summary of what was discussed, which I think is very important for any developer. I'll put a link to this post. So build set targets and then optimize people skip the set target, but then it's emotional decision, make it objective and define what is good enough, right? So even in my experience a lot of times people are not setting targets and running proper evolves and then as a result the system that you build is just based on whether it looks good, which is not an objective measure. To improve any system you need to have quantifiable measures and those are the measures that you're going to set as targets and improve upon. So evolve driven development. Don't ship a component until you have an evolve to test it. This is especially important if you're building a rack system. So for example, RAG has two components. One is retrieval, the other one is the generation component. A lot of times people just look at the final answers without looking at what chunks were retrieved. But you actually want to measure whether you are retrieving the correct chunks, whether your re rank is getting the proper chunks, and then also whether your generation step is generating proper responses. So it seems like OpenAI also recommends just to do end to end evolve and you want to measure at each and every step of your system. Now if you want to make your apps more accurate you need to do prompt engineering, rag fine tuning, do it all lean into meta prompting. O1 actually can help you a lot with this. Even in their playground now they have a tool which will do meta prompting for you. You provide a prompt and the LLM will improve that prompt for you. Some of the techniques that they recommend to try and it's retrieval with cosine similarity, fine tuning, chunking, re ranking, classification step prompt engineering, tool usage and query expansion. Now the other components are latency and cost. These are very important components when you are building any system. If you want to learn more about those, I will post a link. The second interesting aspect is structured output. The way they are doing it is extremely interesting. So if you talk about the normal text generation from LLM it is pretty flexible. Normally the model can pick any word or token at each step. This is why normal text output is more fluid. But when it comes to structured outputs, you need a lot more control. For structured outputs like JSON, we don't want the model to pick any words. Instead it needs to follow a strict format. To do this, we limit the words token it can choose and this process is called the masking of the tokens. So how does the token masking works? So at each step the AI predicts the probability of next word and we then mask or block words that don't fit the required format. Like ignoring a word that doesn't belong in a JSON object. These masks have to be updated at every step of the generation process so the AI knows what words are allowed next. That's why you probably have seen that usually the first output or structured output is pretty slow, but the second and subsequent outputs are usually fast. The first time you request a structured output, the system needs to build an index or a sort of map based on the structured you are asking for, like a simple JSON schema. Now this is a detailed process where the model checks all the possible states and builds up the rules right? So this involves setting up a grammar and parser that ensures that everything fits the structured format, which takes some time and that's why the first structured output token is usually slower. It's very interesting to see the amount of details they were sharing and which is actually pretty good because this gives the developer ideas on how to build better structured output systems, even for other LLMs well beyond OpenAI. I just wanted to make this video on the demo day some of the announcements that we saw in the blog posts and some of the things that I personally found really interesting. I'll be creating a lot more content on these specific topics including the real time API, prompt caching and fine tuning. If that interests you, make sure to subscribe to the channel. Thanks for watching and as always see you in the next one.