Running MPT-30B on CPU - You DON"T Need a GPU

Do you actually need a powerful GPU to run a 30 billion parameter model? The answer is no. In this video I will show you how to run a 30 billion parameter model with 8K context window on a CPU without needing a powerful GPU. We're going to be specifically looking at MPT 30B chat model that was released a few days ago. In this video we are going to be using a quantized version of the model from the bloke. Now, although you do not need a powerful gpu, but you need a lot of ram. So in this case we're going to be looking at this four bit quantized version for which you need around 19 gigabytes of storage space and you need around 22 gigabyte of RAM. Now keep in mind that this model supposedly has a context length of 8,000 tokens, so it makes it one of the more powerful open source models. The rest of the video is going to be divided into two parts. In the first part I will show you this specific repo and we'll walk you through a step by step process of how to download the model and do inference on top of it. In the second part, we're going to look into the code in a lot more details to show you what is happening under the hood. So let's get started. First and foremost, you need to have CONDA installed on your computer. So first we will create a virtual environment. So I'm going to use conda, create, then dash n and name of the virtual environment. So let's call it MPT 30. And then we need to define the Python version that we want to use. So in this case I'm going to be using Python 3.10.0. So this will create a virtual environment for us. Now it's simply asking me to proceed with installation of all the packages. Now you can type in Y, so that will be yes. Or if you notice there are brackets around Y, so that's the default value. So you can simply click Enter and this will proceed with the installation. Okay, so our CONDA environment is ready to use. So I'm going to use this command, conda, activate, and then the virtual environment that we just created. So when I hit enter, you will see that the virtual environment that we are using has been changed to MPT 30. Okay, so we are all set with our virtual environment. Next we need to go and clone our repo. So we will go back and now you see this green button? So click on this and copy this address. So after that go back to your terminal. Okay, Next I want to change my working directory. So I'm going to type in CD documents and then I have a GitHub folder in there, so I want to go there. Next, I want to clone the repo. So I'm going to type in git clone and then the address of the repo that we just copied. Now, usually what I want to do is I want to put this in a custom name folder. So let's call this MPT 30. If you do not provide a custom name, at the end, it will create a folder with the name of the repo. So let's just hit enter and it will clone everything. Okay, Next let's change the current directory to this new directory or repo that we just created. So I'm going to type in CD MPT. It's supposed to be MPT 30. So you will see that actually the folder in which we are currently working has been changed from GitHub to MPT 30. If I type ls, it will simply show me all the files that are within this folder. Okay, the next thing that we need to do is to install all the required packages which are under this requirements. Txt file. Okay, so I'm going to type in pip install r and then requirements txt and it will go ahead and require install all the required packages for us. Now the installation is done, so let's go back to the repo and see how do you actually run this? So there are basically two steps in order to download and run this file. First, you actually need to run this script Python download model, and then you need to run Python inference PY script in order to use the model. Okay, now in order to download a model, you need to run Python download model PI and it will download the model and put it in a new folder called models. Now keep in mind that is a relatively big file, so it's going to take a while to download. Now, if you do not have a stable Internet connection, I would actually recommend you to go back to the blocks repo and within there, click on the files and versions and then manually download this specific file. Just put it in the models folder. Okay, so after running the code, if you look inside the models folder, you should see MPT 30B 4bit quantized model with a dot bin extension. Now we are all set. Let's go back to Terminal and let me show you how you can actually run this. So in order to ask it questions, you can simply run Python inference py and then wait for the model to be loaded. All right, so once the model is Loaded, you will see this prompt user and you can provide your initial prompt. So let's say I'm going to ask it who was the first president of usa? So we're not testing the model, but I just want to show you how it works. Now when you run your query, it will take a few seconds to generate a response. So it's simply processing your prompt. But once it process the prompt, the generation speed is really good. It doesn't take a while to actually generated. Look at this. Like this is the original speed at which it's generating all the tokens. So it says the first president of the United States was George Washington and he was inaugurated on April 30, 1789. Right. So the speed of generation is pretty good. I'm running this on an M2 with 96 gig of RAM. However it's currently using around 26 gig of RAM. Now according to the book, you need 22 gigabyte of RAM to actually run this, but I would say you probably need a little bit more than that. Okay, in the second part we are going to be looking at the code in a lot more detail and I'm gonna walk you through it, how it's actually working. Okay, so it's simply using Hugging Face Hub to download the model. So let's look at the code, how it's working. First and foremost from the hugging face you need to provide the repo or repo ID. So in this case we were using the bloke MPT30B GGML model. Now this code is going to work for all other GGML based models as well. So you can just reuse this code. Next we are defining the model name. So this is the specific model name that you have to choose. Now if you recall, we downloaded this specific file. If you want to use any of the other ones, just simply provide that file name and after that the name of the folder where you want to put the model. So in this case we are calling it models, but it can be anything and you don't have to create this folder beforehand. The code will create that folder for you. So it's using this download mpt quant quant function and you need to provide the destination folder repo ID model name. And under the hood is using the Hugging Face Hub download function from Hugging Face Hub. Now first simply gets the absolute path of the folder name that you have provided, right? And then use the repo ID file name, the location of that path and then a local directory use symlinks. So basically it will simply create that folder for you if it doesn't exist and download your.bin file in there. Now, very soon we are going to be integrating this into local GPT as well. So you will be able to run GGML based models in local GPT. Now if you do not know what local GPT is, I would recommend watch this video. Okay, next let me walk you through the inference pycode. So in this case it's using the C transformer package. And from the C transformer package we are importing automl for Causal LM as well as auto config. Now first we're using the generate config class. So these are different parameters that we need to provide in our to use the model. Next is the prompt template that this model uses. So you need to provide your prompt in a very specific format. First you need to have a system prompt, then the user prompt and then it will generate a response. So that's the assistant prompt that you get as a result and it uses these special tokens in order to delineate between these different parts. Now you can go and look at the block repo to see the actual prompt template that is being used. Okay, next we have a generate helper function that we are going to come back to in a little bit. But before that let's look at the main script. So first and foremost you are defining the configuration and you are getting those from the pre trained model. So you simply need to provide the GitHub repo ID and then auto config from pre trained. So it will simply look at the configuration of this specific model and get all the configurations. Next we want to use the auto ML auto model for causal lm. So here, since we're using a pre trained model, so you need to provide the path of the downloaded bin file, then the model type and the configurations that we actually got from this specific repo. So this is basically setting up your LLM. Next we need to provide the system prompt. So the system prompt that is being used here is a conversation between a user and an LLM based AI assistant named Local Assistant. Local Assistant gives helpful and honest answers. So this is the system prompt that is being used. Okay, after that we are simply defining all the configurations that we will need. Now here you can set the temperature and some other things that you want. Now we set stream to true and that's why if you notice in the earlier example it was generating the response word one word at a time, right? And the maximum number of tokens is basically the number of tokens that are going to be generated. So you can adjust this if you want. And we are using threads, so we are simply looking at the CPU number of CPUs and half of those are basically used as threads. And after that you actually have a special token which is part of the prompt. Okay, so these are basically all the configurations that we need to set. And here is how we want and the user and the model interaction to look like. So within the for loop this is going to run forever. All right. There is absolutely no check. So maybe a better way is to do something like this. If user underscore prompt equal to equal to exit. This is what I'm doing within the local GPT code. So you simply break this, right? Otherwise you accept an input from the user. Then you take the LLM that you just created right along with the configuration, the system prompt and the user input and you go to this generate function. Now within the generate function it's accepting the LLM configuration system prompt and the user prompt and then is calling the this format prompt function to format in the desired format, right? And it will just return you the response. Now if we go back to our while loop so you get the generated response and then you simply show that one word at a time. Now this is how you can run a 30 billion parameter model on a CPU. And as I said, we are going to be integrate this as a part of local GPT so you will be able to use quantized models on your CPUs. All you need is a good amount of RAM on your system. I hope you found this video useful. If you are interested in learning more about LLMs or other field of generative AIs, I would recommend you to check out the Discord server. It's actually a growing community of like minded people who are helping each other out. Link is going to be in the description of the video. If you are working on a product and would like to consult with an expert, check out description of the video for my services. Consider liking the video and subscribing to the channel if you found this useful. Thanks for watching and see you in the next one.