Mistral MoE - Better than ChatGPT?

We finally have the official release of Mixture of Experts or MOE from Mistral AI and they made some really big announcements. There is a version of this open Access model that is able to beat ChatGPT on all the benchmarks by a huge margin. They announced the release of not only one but three different models. They also announced the release of their embedding endpoint which you can use in your own RAG applications. They also announced their own platform just like OpenAI. So now you can access their models through an API. But there's another company which is offering better prices for Mistral AI's models. There is already a controversy regarding their terms of use. We are going to talk about where you can access these models for free as well as some paid API endpoints. We have a lot to cover in this video so let's get started. Over the weekend Mistral AI just dropped the weights of their Mixture of Experts model without any inference code whatsoever. But today they announce the official release and they are calling it Mixtral of Experts which is a high quality sparse mixture of Experts or moe. But this is not the only model that they released because we are going to look at the other models later in this video. Now they're calling this Sparse Mixture of Experts model or SMOE with open weights. So this is not an open source model, rather it's open weight or open Access model. And that's why they say license under Apache 2.0 Mixtral outperforms LLAMA2.70 bill on most benchmarks with 6 times faster inference and we are going to talk about why it's able to get faster inference. It is the strongest open weight model with a permissive license and the best model overall regarding cost, performance trade offs. In particular, it matches or outperforms GPT 3.5 on most standard benchmarks. So we are going to look at some of the benchmark results which are extremely impressive for a model of this size. It has a context window of 32,000 tokens, so it's much larger compared to the original Mistral 7B. Now it is multilingual so it has support for English, French, Italian, German and Spanish. There is no support for any of the Asian languages. They are highlighting its strong performance in code generation and it can be fine tuned into instruction following model. That achieves a score of 8.3 on Emptybench, which is extremely impressive. Before looking at the architecture of the model, let's talk about the performance and how does it compare to Llama 270 billion model as well as GPT 3.5. Well out of the seven standard benchmarks on which LLMs are usually tested, this new MOE model from Mistral AI is able to outperform GPT 3.5 on five of them, which is I think the first time for an open weight or open access model. Now the two benchmarks on which it lags behind GPT 3.5, these vino grande benchmark, but you can see the difference between them is very small. Even on this benchmark the difference is negligible because GPT 3.5 is able to achieve 81.6% accuracy whereas this Mistral MOE is able to achieve 81.2% on the Vino Grande benchmark. On empty benchmark the difference is even smaller. So it's 8.32 for GPT 3.5 and 8.30 for MOE from Mistral. But there is a version of this model which is able to beat GPT 3.5 by a large margin and we're going to look at that later in the video. Now compared to llama 2,70 billion parameter model, the performance of this MOE model is very strong. Now let's see how it's able to achieve this performance. In the blog post they say Mixtral is a sparse mixture of expert. Network is a decoder only model where the feed forward blocks picks from a set of eight distinct groups of parameters at every layer per every token a router network chooses two of these groups. So at a time for every token we're just using two of the experts out of eight experts and then it simply combines the prediction to generate a final prediction. So because of this technique you have an increase in the number of parameters a model is seeing while controlling the cost and latency. Now if you look at it, the mixture of expert has a total of around 47 billion parameters. But for each token it's going to be using around 13 billion parameters. And this is because that is just using two experts for making prediction on each token. So as a result you'll get better performance but improved speed because it's using a fraction of the parameters that are available in the network. Now for the first time they have given a glimpse of what the training data looks like, but there's no not much information. So this states Mixtral is pre trained on data extracted from the open web. We don't know what was the end time of data collection and they trained both the experts as well as the router network simultaneously. Now as stated above, since it's using a subset of the network parameter for each token, so the inference speed is going to be much higher compared to something like llama 2 models because those are dense transformer models and they use all the parameters in the network to make a prediction. So these sets of plots simply shows the increase in performance relative to the inference budget, which is speed of prediction. So almost for all the tasks that you see, including the MMLU data set, different knowledge and reasoning tasks, comprehension, mathematics and code generation is able to perform better compared to Llama 2 family of models, but at an increased speed. The numbers that you see in here are for the base model. However, if you want to use these models you need to fine tune it and convert it to an instruct or chat version. That's exactly what Mistral has done in here. So they also release instruct version of these models. For that they used supervised fine tuning and then did alignment during Direct Preference optimization or dpo. DPO or Direct Preference Optimization is a alternative to RLHF or reinforcement learning through human feedback. Now in this case you don't really need human feedback, rather you use another model such as GPT4 to give you what responses it prefers or other responses. So this is an easier way of aligning the model because you can really automate the whole alignment process. It seems like the Instruct version is kind of uncensored because here they put this note. Mixtral can be gracefully prompted to ban some outputs from constructing applications that required a strong level of moderation and they have provided an example in here also. A proper preference tuning can also serve this purpose. So if you want to include some sort of moderation, either you can build a system around it or you can further fine tune it for your own preferences, you can run this model locally. You will need around 35 gigabytes of VRAM to load and run this model. Before showing you how to access and use this model for free, let's look at some of the other models that Mistral AI has released. Along with this MOE model, they also announced the release of their own platform where you can apply for access. They are releasing three different models that are going to be available on this platform. So the first one is what they're calling Mistral Tiny. So this is going to be their most cost effective endpoint. Currently serving Mistral 7B instruct version 0.2. This is a second release of their original Mistral 7B model. This is English only and it's available for download on Hugging Face. If you go to their Hugging Face account, it's available under the name Mistral7B instruct v2.0. You can actually fine tune this model on your own data set if you want. I have a video on that. I'll put a link to that video in the description. The second model is what they're calling Mistral Small and this is the new or mixture of expert model that we looked at. And as I said, it supports English, French, Italian, German and Spanish and it's comparable to GPT 3.5 on almost all the benchmarks. This is also available on Hugging Face. You can download the model and experiment with it. It's already integrated into Transformer library from Hugging Face. In an upcoming video, I'll show you how to run this. Their most powerful model is what they're calling Mistral Medium. Just pay attention to the medium. They probably have a larger version and I think that one is going to be crazy. So according to a blog post, this is their highest quality endpoint currently serving a prototype model. So seems like it's still under training and they probably are going to be making more updates to it. This is currently among the top serviced models available based on standard benchmarks and the style. The benchmark numbers are crazy. Now this also supports English, French, Italian, German and Spanish. And on the empty benchmark data set is able to achieve a score of 8.6. For comparison, they have provided this table and they're comparing it to GPT 3.5 Mistral Small, which is the MOE model and Mistral Medium. We don't know if it's a mixture of expert or not, but most probably it is. And if you look at all these numbers, it blows GPT 3.5 out of orders on all the benchmarks and not by small margins. These are crazy margins for standard benchmarks. I think the credit also goes to the Mistral AI team because they're not comparing this to GPT4 because GPT4 is a whole different beast when it comes to LLMs. So even though Mistral Medium is the best performing model, keep in mind they have not released the weights. It's only available through the API endpoint. We are going to also talk about some of the controversy behind their terms of services later in the video. Now, apart from these LLMs, they are also announcing the release of the embedding endpoint. So this is going to be Mistral Embed, that's what they're calling it. It's an embedding endpoint which is going to have a dimension of 1024. So that's the vector size. It's able to achieve a score of 55.26 on the massive Text embedding benchmark. Now, this is not the best score out there, so if you go to the leaderboard for embeddings, it is going to be probably, I think, somewhere between the 57th and 58th position. But just having an embedding model itself is, I think, great. Because now you can build RAC pipelines by using both the embedding models as well as LLMs from Mistral AI. But they're not releasing the model itself. It's going through to be available just through the API endpoint. Now, in terms of the API specifications, so they say API follows the specifications of the popular chat interface initially proposed by our dearest competitor. So here they are referring to OpenAI. So seems like the API endpoint specifications are going to be following OpenAI's standards or their format, which is actually pretty great because now you can simply replace OpenAI API with an API from Mistral AI. So for developers it's going to be a lot easier to switch between these different vendors. Now let's talk about some of the pricing. So if you're using Mistral tiny, small or medium, as well as the Mistral embed model, they have the pricing available. This is relatively cheaper compared to GPT 3.5. However, if you're using the tiny or small model, I would not recommend using their services. I would rather recommend you to look at something like Together AI. They are offering much cheaper prices for both the tiny and small model. But here you will not be able to access the Mistral media model because that is not available apart from just the API endpoint through Mistral AI. Now here is a comparison with the GPT 3.5 Turbo. If you look at it, this MOE model is much cheaper to use compared to GPT 3.5 through the API from Together AI. And in terms of the performance on the standard benchmarks, it's on PAR with GPT 3.5. So now we actually have a legitimate alternative to GPT 3.5, which is absolutely amazing. And all of this has happened within one year of the release of Chat GPT, which is absolutely crazy for the open source community. Now, if you're using the API endpoint, you just need to be careful about one thing in their terms of use or terms of services. This was highlighted by far L on Twitter or X. According to this, you are expressly prohibited to use the outputs or any modified or derived version of the outputs to directly or indirectly reverse engineer the services and or to develop directly or indirectly competing services. So seems like from the outputs, most probably just the API endpoint, you can't really use that to retrain another model that is going to compete with Mistral AI. So we will probably need some lawyers to shed more light on this, but it's in their terms of use so make sure you are fully aware of this. If you are going to train a model that is going to compete with Mistral AI, you can run this model locally or more practical is going to be running this in cloud because you need a substantial amount of VRAM to load and run the model. I'm going to show that to you in the next video. I also integrated this as a part of the local GPT project and I have seen some really impressive results in terms of integrating this as a part of RAG Pipeline. Hopefully in an upcoming video I will demonstrate that as well, but you can play around with the model for absolutely free if you want. So Hugging Face integrated this as a part of their hugging chat platform so you can select Mistral AI mixtral of experts from here and then you can play with it. It's a really impressive model and it deserves its own video. So I'm going to make a video testing the model, but here's a prompt that I usually use and so far I haven't seen and if the open source model give me correct responses, the prompt is how many days will it take for a pond to be half filled with lilies? If the number of lilies double every day and initially it takes 48 days for the pond to be completely filled. Once you generate your answer, critique it to answer to ensure its correctness and on the first try it states that it would take 47 days for the pawn to be half filled with lilies and then it goes on and give us the reasoning. So the number of lilies doubles every day. This means that on the last day when the pond was fully covered, that is going to be day 48. Just one more day earlier, that's day 47 would result in half of the pawn being covered because the lilies courage has turned doubled from the previous day. This is pretty nice reasoning. Here's another famous prompt. If 10t shirts laid out in the sun takes five hours to dry, how long does it take? 20t shirts? Again, most of the open source models that I have tested have trouble figuring this out, but it does it in the first try. So it states given that they are adequate drying space and sunlight, it would also take five hours to dry 20t shirts and here's why. And then it explains the reasoning behind the answer and it does A really good job. Here's another tricky one. There is a room with three murderers inside. Another person enters the room and shoots one of the three original occupants dead. Nobody else enters and leaves the room. How many murderers are in the room? And again it's able to figure this out. So it states four murderers are in the room, the initial three for being murderers and the fourth person who just shot one of them because they also became a murder by committing an intentional killing. And here is a response from ChatGPT GPT 3.5 so it states there are still three murderers in the room. When the new person entered and shot, one of the three original occupants stand dead, the person became a murderer as well. So there are still three murderers in the room. As you can see just from these few example tests, this is a very capable model for its size because effectively it acts like a 12 billion parameter model. Now thanks to the open source community we already have some fine tuned version of this MOE model. So for example Synthesia MOE v3CE. Now this was fine tuned on the unofficial released version. So this was based on the weights that were initially released as a torrent file, but the after the official release we are going to have I think a version tomorrow. Anyways, it's a very exciting model. Great to see that we have open access and open weights model. Now you probably noticed that I did not call it an open source model because they just released the weights. We don't really have the training data, neither the training code. And for a model to be a completely open source model or open source open source LLM, we need to have access to the training data, how it was trained, so the training code as well as the trained model which are the weights. So hopefully we can still use this as a community to build better models because at least the weights are open access. I hope it gave you a quick overview of what Mistral AI has released and hopefully in the near future we might have a model which is on par with GPT4. So 2024 is going to be very exciting year for open source large language models and I can't wait to see what the future holds for us. Thanks for watching and as always see you in the next one.