Mistral MoE: Benchmarks, Instruct fine-tuned version, 4GB vRAM and Free Access on PerplexityAI

Yesterday Mistral AI released their mixture of expert model and a lot has happened since then. Now we have benchmark scores from that model and they are absolutely amazing. Perplexity AI is now hosting this model with free access to everybody and there is even an instruct fine tuned version of that model. Thanks to the awesome open source community and the cherry on top, we might just need 4 gigabytes of VRAM to run this model. We have a lot to cover in this video so let's get started. Let's first talk about the benchmarks. So here is a tweet from Anton who is very active in the open source LLM community. After playing with this new Mistral model for the last 24 hours, pretty sure if you go through fine tuning and RLHF you would get a model that is better than GPT 3.5 and it's going to be a local model. Now this is a very bold statement to say the least, but he's able to back it up with numbers. These are the performance on different benchmarks. This is based on the work from Bjorn who has an excellent account on Hugging Face. I would recommend everybody to check out different models that he has. He specifically works on German models and there are a whole bunch of data sets over there as well. Just keep in mind that these results are not for a fine tuned model. The benchmark scores that you see in here are for the base model. Later in the video we are going to look at a fine tuned version of the Mistral mixture of expert model. Now how good these scores are? Well, for comparison we can look at the scores of GPT 3.5. First we're going to look at this. Hello SWOG data set. On this benchmark, the new Mistral model is able to achieve a score of 0.8661. If you look at GPT 3.5 on the same data set it was able to achieve 85.5% with 10 shot helice. Swag is a common sense reasoning around everyday events dataset. This is not just the case with the helicewag dataset. For the rest of the dataset we see a similar pattern. So for example, for this famous MMLU data set, this mixture mixture of expert model is able to achieve 71.73%. GPT 3.5 is able to get to 70% with five shot learning. Now we don't know whether for this mixture of expert models they used 5 shot learning in testing or 10 shot. So that information is not yet available. Another thing, these are not official scores so this is an open source community member doing the benchmark scores. We are still waiting for Mistral AI to release the official score. So it will be interesting to see what the official scores for instruct fine tuned model look like even for the GSM 8K data set which is grad school mathematics question. The performance of this new model and GPT 3.5 are very similar to each other. This is huge if you're able to further validate these results. If you want to run this model, you have a couple of options. So yesterday I showed you this instance from Replicate where you can experiment with the model. But now Perplexity AI, which is an AI based search engine, also added this new model to their labs. So now you can come to the list here and at the end of the list you're going to see this mixture 8x7B. This is the mixture of expert model. Now keep in mind the way you prompt this model is very different than the way you prompt an instruct or chat model. The base model are what we call the next word prediction models or sentence completion model. So in this case you can provide a sentence or part of a paragraph and then the model is going to start completing that for you in case of chat or instruct models. In that case you provide a question and the model is going to answer you. And this behavior comes from the way these models are trained. So base models are trained on a large corpus of language data without giving it much details about what a question answer looks like. However, this is the first stage of training a language model. In the second stage, you simply take a base model and then provide a prompt response pair and you retrain the model again so that whenever somebody asks a question, the model is going to generate a response related to that question. So for example, in order to use a base model, you might want to do something like this. So here is an example sentence. The capital of France is. And now the model is going to complete this sentence. As you can see, the model answered or completed that sentence saying capital of France is Paris. And then it kept generating more text that may or may not be relevant to what we are talking about. So for example, in this case it had these special tokens and then it continued stating that a famous French painter was Claude Monet. And again some special tokens. So this is coming from the training data. Now as you can see, the base models are not really useful for practical applications. You want to have an instruct fine tuned version of the model so that you can use it in your own specific applications and that's why we are going to be looking at an instruct fine tune version of the mixture of experts model from Mistral. Next, before looking at the instruct fine tune version of this model, here's a very interesting post from Tim who is the creator of the bits and bytes package. If you don't know what bits and bytes is, it's a lightweight wrapper around Cuda custom functions in particular 8 bit optimizer and it really helps you in running these large language models on Nvidia GPUs and thanks to this package you are able to load these LLMs on CUDA GPUs in 4 and 8 bits. So here's the post. I think we can compress this model down to 4 gigabytes of vram. I worked on MOE which is mixture of experts compression before and sparsification was very helpful for MOEs. Now the most important aspect of this post is the sparsification of MOEs. They behave very different to the normal LLMs which have dense layers. In this case, at one time only two experts are active for any one token prediction. So that means you don't really need all the other weights if you're making prediction for one token because you just need two different experts and that actually enables you to do the compression. Now according to him, I have already implemented this compression in bits and bytes so code not optimized yet. Could somebody test this implementation before we integrate in it into hugging face transformer? Later in the post he talks about how this can be done. So for example, you can replace the MOE MOE layers in this mixture of expert model with sparse linear layer and that acts exactly like any other four bit or eight bit bits and bytes layer. I'll put a link to this post in the video description if you're interested. Go over this. It has a lot of information on how these these models can be made more sparse and hence reduce the VRAM requirement. All of this work is driven by the open source community. In the last 24 hours the last update that we had from Mistral AI was just that torrent link. They have posted the model weights with absolutely no other information on how to run this, what are the benchmarks and how does it perform. So all of this is purely based by the work the open source community is doing. So that brings us to this. Now we have an instruct fine tune version of the Mistral's mixture of experts model. This is coming from an organization called Disco Research. So let's go over how they fine tune this mistral mixture of experts model According to the description, this is an experimental 8 by 7 Bill Moe model based on the Mistral's 8x7B model. This model is based on experimental code converting the model weights to hugging face format and then enabling transformer based inference. Then they further fine tune the base model on three three different instruct data set. So one is Synthesia, the other one is Metamath, Q and A and the last one is Kebara dataset. They also provided some results for this Instruct fine tune version of the moe. But with this statement this model is still an early alpha with experimental code and we cannot guarantee that all these values are correct. Actually nobody can guarantee because we don't have access to neither the training code nor the inference code. Now in terms of the benchmarks results, I think it's very important to compare those to the results that we already saw for the base model versus this Instruct fine tune version. So if you look at the LS swag numbers, it's around 86% here, 80. 86.61% here. So it tracks pretty good. Even the MMLU score is again very close to the base model. Now for Some reason the GSM 8K results from the fine tuned version of the model are really bad compared to the base model in here and according to the author they have no clue why. So it might be related to the fine tuning data set that they use. Although there is a math related data set, but again it's all experimental. I think the only way we can actually measure the performance of this model is if Mistral AI is kind enough to release both the training as well as the inference code. Now how exactly you run this model? So basically they trained it with the chat ML format. So you need to provide a system message, then the prompt from the user and the assistant will generate the response. Here's the inference code using the transformer package from hugging face. So again you need to provide a system role. So this is going to be your system message then user input or prompt and you will be able to generate a response. Now I personally have not tested this yet. I will be testing this pretty soon to see if we can get any coherent responses out of this instruct fine tune version. These were some really great developments in the last 24 hours since the release of this model. By Monday we might have quite a few more fine tuned versions. So I'm really looking forward to those and also looking forward to when Mistral actually releases their training as well as inference code as well as their own internal benchmarks results. If you don't know about this whole Mistral drama, I created a video yesterday on this where I covered the model release and everything else related to that. If you haven't watched that video, I'll put a link to that video. Go watch that. That will give you a lot of context. I hope you found this video useful. Thanks for watching and as always, see you in the next one.