Mixtral MoE on Apple Silicon is Here, thanks to MLX

The MLX framework from Apple is getting a lot of traction because it enables you to run LLMs apart from other machine learning models natively on Apple Silicon. The Apple machine learning team is working on putting support for more and more models. In this video I'll show you how to run the Mixtral of experts from Mistral AI on Apple Silicon using the MLX library. You can actually run LLMs on Apple Silicon using libraries like Llama CPP as well as the Transformer package from Hugging Face. However, with MLX now we have native Support for these LLMs in Apple Silicon. The only thing is that you need to convert model weights into the numpy format. However, thanks to the MLX community on Hugging Face, we have pre configured weights for several LLMs. These include Llama 27B, Mistral 7B, Zephyr 7B, and even 87B, which is the mixture of expert from Mistral AI. There's even support for Pi2. But in this video I'll just show you how to run the Mixtral 8.7B model on Apple Silicon using the MLX framework. The process of running the other models on MLX is going to be exactly the same. In a previous video I talked about MLX in detail and covered its features. I'll put a link to that video. Keep in mind that you will not only need a lot of VRAM in order to run this model, but you will also need a lot of hard disk space. The files that you see in here actually comes out to be around 96 gigabytes, so you will need at least 96 gigabyte of hard disk space in order to just download the weights of this model. First we're going to create a new virtual environment and we're going to call this mlx. For creating new virtual environment I'm using CONDA so Condo create dash n and for this specific use case we're going to be using Python 3.10. Next we are going to activate the virtual environment using conda. Activate mlx. Here you can see that we are in the new virtual environment. After this we need to install the MLX package, so we are going to be using the PIP install MLX command. Don't worry, I will provide all these commands in the description of the video. In my case, as you can see I have already the package installed my machine. Since we are going to be downloading the model weights directly from Hugging Face, that's why we will need to install the Hugging Face hub as well as Hugging Face transfer packages. Again, you can see that I had already installed these packages. Next, we are going to clone the MLX example repo using this git command. Once the repo is cloned, we need to move to that folder. So we're going to use the Change directory command and then provide MLX example folder path. If I type LS command, this will list all the files present in the folder. Next, we need to download the model weights. But for that we need to set this one environment variable. So hugging face hub transfer that will be set to one and we'll run this command. The next command that we're going to run is this hugging face CNI download. Then you need to provide a folder path. So we are calling the folder mixtral87b instruct and it's going to download model weights from the MLX community on hugging face. Now, this process is going to take a while because it has to download about 96 gigabytes of files. Now, in order to run the model, you need to make sure that you are in the MLX example folder. Then within here we have the LLMs folders, and within the LLM folders we actually have the Mixtral folder. So you need to make sure that you are within the MLX folder. And here, as you can see, we're going to be able to see the LLM folders. So inside the MLX folder, here's the command that we are using. We're using Python and then running this file, and we're going to provide the folder or path where we downloaded the model. So if you run that command, you're going to run into this issue because the code is not able to find config. Now, the reason is that within the mixtral87b folder, we need to rename this parameters.JSON file to config JSON. So I simply renamed that file to config JSON. Now let's try to run this again. Okay, so it's loading the model from disk. That's a good sign. After making this change, we can run this command again. However, in order to use the LLM, we need to provide a prompt. You need to provide this extra flag dash dash prompt and then your prompt. In this case, my prompt is what is mixture of experts? And let's run this and see what the output is. Later in the video, I'll also talk about the memory usage of the model. Okay, so it took quite a while, around five minutes. It's still generating the responses, but at least it generated this response so far. So it says mixture of experts and it's still processing the data and it is using around 93 gigabytes of RAM out of my 96 gigabytes of RAM. I'm just curious how long this is going to take to generate a whole answer. It's been more than 15 minutes so far and the model hasn't generated the full response yet. So far we got Mixture of Experts is a machine learning technique used for modeling complex data distribution. It is the type of ensemble learning method that combines the prediction of multiple expert models, each of which is specialized and we are still waiting for the model to generate more tokens. For some reason it's not really using the GPU full capacity. Usually when I run quantized LLMs with llama CPP on my M2 the GPU usage is pretty high compared to what we see here. Alright, so seems like I was a bit too ambitious with this studio. Well we have some more tokens so it says specialized in different parts of input space. So as I was saying, I'm probably too ambitious with running MOE on my M2. We got some more tokens out of the models, so now it states each of which is specialized in different parts of the input space. I was a bit too ambitious to run this model on my M2. Now keep in mind that this is not the quantized version of the model. I haven't seen support for quantized models in the MLX library yet. Probably Apple is going to add that soon. Now in terms of the actual memory usage, so out of 96 gigabytes of physical memory that I have on this system, the model or the system is currently using 93 gigabytes. So it's not just the model. There are a whole bunch of other softwares running, but I think a major portion of this is occupied by the model. Theoretically you can run the MOE on an M2 Max with 96 gigabytes of VRAM, but you might have to wait for a while in order to get a response. Currently we are closer to 20 minutes and it hasn't completed a whole paragraph yet. Now the same hardware that I have, it has no issues running Mistral 7b, llama 7b, or even the 13 billion parameter models in unquantized format. But this mixed rule of Expert is I think a bit too much for my hardware. Okay, we got some more tokens as well. Now it states the basic idea behind MOE is to divide and we're still waiting. But the basic steps in order to run a model using MLX framework or package are going to remain exactly the same that I showed you in this video. This is taking way too long here. So what I'm going to do is I'm going to post the final output of the model in the description of the video. I'm looking forward for MLX to add support for quantized model. I think that will make it a lot more powerful and useful. Let me know if you are able to run the moe on your M3 max. I would love to see what type of results you are getting out of this model. Okay, so we have some more tokens and now the response is the basic idea behind MOE is to divide the input space into several regions and assign a. I don't know what it's going to say after that, but I hope you found this video useful. At least you learned how to run LLMs using mlx. Thanks for watching and as always, see you in the next one. Thanks for watching.