Mistral - MoE | The Most Unusual Release & How to Run

Mistral AI just released a new model in the most unusual way possible. From their official Twitter handle, they simply tweeted a link to a torrent file. The torrent contains model weights. There is no inference code or no model card whatsoever. Okay, initially everybody was just confused. Here is the name of the file within the Torrent file. It's Mistral 8x7B 32K sequence length. Initially some people thought that it might be a multimodal model, but then it turned out to be a mixture of experts just like GPT4. So there are speculations that GPT4 is also a mixture of experts. Now as I said, there is no official press release on their company website and nothing on their hugging face account as well. So everybody is just scratching their heads of how to actually run the model. So later in the video I'll show you how to run this or experiment with the model itself, but that is going to be an unofficial implementation. The only news I could find related to Mistral was this from the Financial Times it states, French AI startup Mistral secures 2 billion euros valuation. Seems like they just completed a 400 million euros round, which is kind of crazy because the company itself is just 8 months old. But I think apart from Meta, this is the second biggest company that is driving all the innovation in open source large language foundation models. Here is a tweet from Andre Karpathi regarding the model architecture and he also referenced to this GitHub repo which is likely related to the model, but we are not sure. And in the tweet he took a stab at Google's Gemini release, so it states, oddly absent an over rehearsed professional release video talking about a revolution in AI. Now he is referring to this video that was shown during the release of Gemini, but it's not exactly the way it looks like. It was not real time. There was a lot of prompting involved and the model was shown a sequence of images. Okay, more on that in another video. So here are the official contents of the torrent file that was shared and I must say it's an absolutely amazing marketing stunt because everybody is just talking about Mistral right now. Andre Karpathy in his tweet was referencing to this GitHub repo. It's called Mega Blocks and this is maintained by the Mistral AI team. Mega Blocks is a lightweight library for mixture of experts training. The core of the system is efficient, dropless, moe or mixture of expert. Now we don't know if exactly this approach was used to train this model, but since this library is maintained by the Mistral team. There is a high probability that they use the same or very similar approach. There's a paper related to the Mega Blocks project called Mega Blocks Efficient Sparse Training with Mixture of Experts so I'll put this in the video description if you guys want to read this, let me know in the comment section below if you want me to create a video on explaining what mixture of expertise is. So you might be thinking we have the model weights, can we actually run the model? And the answer is kind of yes. So we already have hacked code that is able to run the model. Now I'm going to put a link to this GitHub repo in the video description. According to the author of the repo, extremely hacky implementation of Mistral 8 times 7B. Now it states Mistral dropped the new MOE model this morning. This is an attempt to hack original Llama code base to load it and then implementation is very naive and slow. Now in order to run this you will need around 160 gigabytes of VRAM. But don't worry if you don't have that much compute. Later in the video I'll show you how to test this implementation for absolutely free. Now the author goes on to some details of how he has implemented and then there is this warning. There is no official reference model code. This implementation might be wrong. At least the generation looks coherent, which is a good sign and I totally agree with him. Now in case if you have that much compute at hand, you can actually run his code. So simply download the model weights from the torrent file or there is now even GitHub or actually hugging face repo where the model weights are available. So here's the link to that repo. Now on the hugging face repo, the username is very interesting. Someone 13574 that's a bit unusual. And in order to run the code so you will need to use python example test completion PY file and then path to the model weights that you consolidated from either hugging face or from the torrent. The model itself is a base model, so it's not an instruct fine tune model. So essentially it will simply predict the next word. So if you give it a sentence it will complete the sentence. But if you start asking question it's not going to give you like question answer pair, it will simply generate text. So that means we will actually need people to fine tune the model on their own data sets. And I think a lot of people are going to be doing that once the training as well as the inference code is available. So here are a sample output from the model. So for example, here's the prompt that starts Mistral AI is accompanied that and then this is the output from the model. It says provides a platform for building, training and deploying AI models. The platform offers a variety of tools and services that can help developers and data scientists build and train AI models. Now in this case the model is hallucinating and we are not even sure that if it's actually the right implementation. Now he also tested it on a few other examples. Now at least it's generating coherent text, so it seems to be working. But what if you want to test it yourself? Well that's also possible. There is another X or Twitter user by the name Natro. He actually took the code that we just saw and put this on Replicate. Now before testing the code, a couple of things from the author. So in this tweet he says keep in mind this is this model is using an experimental Pytorch implementation, so it could be making mistakes. Also it's a base model, as I said, it's not an instruct fine tuned version yet Another issue is the context is hard coded to 512. Now trying to hotfix right now. If you run into any issues making predictions with the latest versions, use this version. So he actually pointed out a version in here. Okay, so here's the unofficial instance running on Replicate and according to it, Mistral's AI new mixture of expert base model for text generation. Now he provided an example prompt in here. Again as I said, it's a text generation model. So the prompt is here are the top 10 useful Hindi phrases for your upcoming trip to India and then it's like number one. So the model is going to start completing this and moving forward. Now max number token though those are fixed, you cannot change it, at least in this demo you can I think play around with temperature and top P. Now here is the response. Now the completions are actually really good. So for example the first phrase is Namaste which actually means hello. Then the second one is and it does mean how are you? And is used to ask about somebody's well being. This is correct. The third one is and it's also correct. I want to listen to you. So I would say the completions are actually pretty good. I don't know if the Hindi text in here, the transcription is actually right or not, but these are actual phrases in Hindi or Urdu. Now for my test I'm going to use this prompt. Here are top five differences between Llamas and alpacas. Now if you notice there is a spelling mistake in differences, but let's see what it comes up with. So here's the completion. Llamas are much bigger than alpacas. Llamas have long necks and ears while alpacas have short necks and ears. Then llamas have longer faces than alpacas, llamas have longer tail than alpaca, and llamas are short, used as pack animals while alpacas are used for their wool. Right? And since it's a text completion model, so rather than stopping here, it actually continues generating more text. So for example, it says what is the difference between a llama and alpaca? So basically it's ask a question, then it generates an answer for that question, right? Then it continues on asking another question, answer, then another question and answer, and so on and so forth until it gets to that 512 max tokens. Okay? So even with this unofficial implementation of the inference code, the model is able to generate coherent text. So that is a really good sign. Now we all, we all are waiting for the official release. Now keep in mind that you will not be able to run this thing on consumer hardware. So it definitely has to be hosted in the cloud where you have enough gpu. Vram. Now, the last couple of weeks have been absolutely crazy to say the least. But we are not done yet. There is more to come, I think. In December, for example, today there was another model dropped called Striped Hyena 7B. This is from Together AI, which is an open source model using an architecture that goes beyond transformers, achieving faster performance in longer context. So I'm going to be looking at that a bit more. There are some innovations and people have been exploring some of the technologies apart from transformers, which most of these models are based on. So it will be very interesting to see what type of architectures people come up with and how their performance is compared to the models that we have today. There is a potential model released from ByteDance or TikTok. So for example, here is one of the staff member of ByteDance. This was a tweet a few days ago. So in that tweet he says uncertain about GPT5 but a super strong model, more powerful than Gemini is expected to arrive any time now, right? Somebody asked him open source and his reply was open model weights. In the last couple of weeks there have been some really amazing and extremely powerful models coming out of China. So I would recommend everybody to keep an eye on those models as well because they have the multilingual capacity, and their performance is on par with something like the llama architecture. So really exciting times. There's never a dull day. I hope you found this video useful. Thanks for watching and as always, see you in the next one.