OpenAI: New Embedding Models, Price cuts for GPT-3.5, Moderation API and More!!!

OpenAI just released multiple new models with substantial price cut. There are two new embedding models. The GPT4 Turbo has been updated and there's new pricing for GPT 3.5 Turbo. And there is a significant performance boost for these new embedding models. They made the announcements in this official blog post title New Embedding models and API updates. And and there is a lot to cover. So there is a new generation of embedding models, there's a new GPT4 Turbo model as well as a new moderation API. And they are substantially reducing the pricing on GPT 3.5 Turbo. So now it's raised to the bottom when it comes to compute for embeddings. There are two new models. The first is text embedding three small, and the second one is text embedding three large. If you're not familiar with embeddings, it's a sequence of numbers that represents the concept within content such as natural language or code. Basically, you can convert your text into these sequence of numbers and perform different mathematical operations. And the embedding models are critical for something like the Assistant API as well as for retrieval augmented generation pipelines. So if you're using anything for chatting with your documents, you are basically using an embedding model. These updates are significant because the last model that was released for embeddings was the text embedding ADA002 and that was released back in December 2022, so more than a year ago. Now the question is, how does the performance of these new models look like? So if you compare the embedding three small model with the existing model, there is a performance improvement from 31.4% to 44% on multi language retrieval tasks. Now this is substantial because we are looking at a performance improvement of I would say 40 to 45%. However, when it comes to English benchmarks, for example, the MTEB benchmark, the performance improvement is not that significant. So it went from 61 to 62.3%. But the major update is to the pricing. The new model is almost five times cheaper than the existing embedding model, which is going to be huge for developers. Now, the existing embedding model is not going anywhere, but they highly recommend to use the newer model because they are much more accurate as well as a lot cheaper. All right, so before looking at the larger version, keep in mind that these are not the only embedding models that you have. There is a leaderboard for embedding models on hugging face. So if you were to compare it this new text embedding model, the smaller version is going to be around on the 16th position if you compare it to the rest of open source embedding models. But I think OpenAI will be able to beat everybody on the pricing since serving your own embedding model will cost you a lot more than what they are offering. The second one is this text embedding three large, which is the larger embedding model and creates embedding up to 3,072 dimensions. Now the most interesting part is you can create embeddings up to this dimension. That means you can also get smaller dimension embeddings as well, and I think that will be critical in some applications. We're going to talk about those in a bit. So this has a stronger performance compared to the existing embedding model as well as the small version because the performance goes from 31.4% to almost 55% on the multilingual benchmarks dataset. On the MTEB benchmark dataset that goes to 64.6%. That means if you were to compare this with the other embedding models on the embedding model leaderboard, that will put it somewhere between third and fourth position, which is pretty a huge accomplishment. Now this is priced very similar to the current embedding models offering from OpenAI, so you could just use this as a drop in replacement for the ADA002 embedding model and get much better performance. I found this to be the most interesting part of the release. There's a native support for shortening embeddings. The ADA V2 model had an embedding size of 1536. However the small embedding model comes in two flavors, so the least size that you can use is 512 and this can go up to 1530. So I think you can choose between these two flavors. For the larger model there are three flavors, 2 56, 1024 and 3 072. Now larger embedding size will give you better performance. However it seems like they compute the embeddings in this size by default, but you can use the let's say first 256 embeddings or up to 101024 embeddings. So it's not really recomputing the embeddings, but you are just using a smaller portion of the embedding size. Now why is this significant? So if you are creating embeddings you need to store them in a vector store. However the vector stores that we currently have available, they support specific strict dimensions. So for example, if a vector store supports up to 10024 dimensions, then you cannot use the larger size. That's why they have the ability to use much shorter size. So this is going to be significant. That means that you can use these new embeddings as a replacement for your own embeddings without worrying about the dimensions. The embedding size or dimension is going to be just a parameter to the API, which will make it a lot easier. Let's talk about some updates to 3.5 Turbo and lower pricing. So they're releasing a new version next week that is going to be available to the users. The most significant update is to the pricing compared to the existing 3.5 turbo. There is a 50% price reduction for the number of tokens that are used as an input and the output prices are reduced by 25% which is pretty great apart from the prices. So this model will have various improvements including higher accuracy at responding in requested formats and fix for a bug. I think there was a bug related to non English language function calls, so that is also being taken care of. They are also releasing updates to GPT4 Turbo Preview. So there's going to be a new model with another name and this model completes tasks like code generation more thoroughly than the previous preview model and is intended to reduce cases of laziness where the model doesn't complete tasks. So hopefully this will take care of some of the concerns and grievances the developer community has regarding GPT4 being lazy and not complete tasks. But we will have to see and again this also includes a bug fix which was impacting non English languages. There is also an update to the fee moderation API that allows developers to identify potentially harmful text. So they're releasing a new version of the moderation API and you can use this for absolutely free. They are updating that platform as well. So now you can see individual API usage and also manage your API keys. They are launching two platform improvements to give developers both more visibility into their usage and control over API keys. This is going to be great because now you can create different APIs for different applications and actually see the usage individually for each API that you have. So the first improvement is developers can now assign permissions API keys from the API keys page. For example, a key could be assigned read only access to power and internal tracking dashboard or restricted to only access certain endpoints. The second one is the usage dashboard. So usage export function now expose metrics on API key level after turning on tracking. As I said now you can track the usage per API key, which is going to be great for developers. So this was a quick video on the latest updates to the embedding models. It's great to see the pricing cuts. This will force other API service providers to also reduce their pricings. So let the the price wars begin. Thanks for watching and as always, see you in the next one.