Updated Oobabooga Textgen WebUI for M1/M2 [Installation & Tutorial]

In this video I will show you how to install the Kuba Booger text generation web UI locally on your own machine. If you want to run open source large language models, this is the best tool that you need to get. It supports a large number of LLMs out of the box and is relatively easy to set up. So in the rest of the video I will show you a step by step installation process and then I will show you how you can actually install the VokUnion 33B, the 1.3 version model. With this. For this video we're going to be specifically focusing on installation for Mac OS with Apple Silicon. If you are interested in the installation instructions for Linux and Windows, I would recommend you to watch this video. Now in this video we are not going to be using the one click installer because I have found it to be buggy when it comes to M1 and M2 processors. Instead, we will be doing a step by step installation to start our installation process. First we need to actually go and clone this repo. So for that go to the top of the repo and click on this green button you will see this link. Now copy this link and open a new terminal. So here let me type in ls. So this gives me a list of all the files and folders that are present in in the current working directory I want to move to the documents folder. So I'm going to type in CD documents. Now in order to clone the repo. So I'm going to type in git clone and then the repo link that we just copied. Now by default if I click enter it will create a folder with this name text generation dash webvi. But I want to put it in a custom name folder. So I'm going to call it textgen and let's click enter and it will start copying all the files from the repo. Now keep in mind for this to work you will actually need to have git installed on your local machine. Alternatively, if you do not have git installed, then you can actually go to the same code button and download the repo as a zip file as here and then unextracted and work with that zip file. Okay, so we're going to go back to our terminal again. I will simply type in clear so that it will remove all the text that we had so far. Next we need to create a new virtual environment. So I will be using conda for this we'll type in conda create dash n and the virtual environment name. So I'm going to also call this text gen, I already have a virtual environment with this name. But. But in your case it's going to be different then, and this is the most important part. You want to define the Python version that you want to use. So for this specific case, I'm going to be using Python 3.10.10. Now you might be thinking, where is this coming from? So this is coming from one of the issues that was created specifically for installation on Apple Silicon. And here this person is recommending Python 3.10.10. So let's go back to our terminal, I'll just hit Enter and it will start creating the virtual environment for you. If you do not have. In this case, it's simply asking me, there is an existing Python version virtual environment with this name, do you want to remove the existing environment? So I'm going to type in yes, and let's wait for it to remove everything and then it's asking me whether I want to actually proceed with the installation of. So I'm going to type in Y and let's start the installation process. Okay, so the installation process is complete and now we need to activate our virtual environment. So we're going to be using this specific code segment, which is conda, activate and then textgen. Now notice here, instead of base it's going to change to textgen. Okay, so I simply clean my terminal again. So let's type in LS and here's the folder that we just created using the git clone command. Now we need to move to that folder. In order to do that we will type in cd, that is Change directory and then type in textgen. And if we type in ls, it will show us the contents of this folder. Now the content of the folders are going to be exactly the same as the repo on the GitHub. So now let's go back. Next we need to install all the required packages and they are here in the requirements txt file. So in order to do the installation for that we will actually need to type in pip, install dash r and requirements txt. Actually, let me show you one more thing before this. So sometimes when you are working with virtual environments, it may not pick the right virtual environment. So you can check which Python you are using, so you can type in which python and then it will tell you the path where the Python virtual environment is located. So in this case we are using the right one, so it shouldn't be an issue. However, I would recommend to use this Python M right, and then pip install R and the requirements txt so in this case it will make sure with this command that it's using the corresponding virtual environment Python. So let's wait for it. So it's simply downloading all the required packages and installing them on our local machine. So the installation is complete. However, we need to do one more thing. So we actually need to go back and install a specific Pytorch version. This is the nightly version that works with Apple Silicon M1 and M2 processors. So we're going to go back to our terminal and type in the command that we just copied. So I'm going to just paste it and hit run. Now keep one thing in mind, that in this specific case, the virtual environment manager being used is different than conda. So just skip this step and install it using conda. Okay, so the installation is complete. Now we are all set to run the script. So I'm going to type in LS again. Now there is this specific file that we need to run and that is the server py. However, we need to make one more change before running the server. Now, to get the most out of the hardware that I have, we will need to set number of threads. So I have a M2 max. That's why I'm going to be setting threads to 8. When I'm running the server py file for your specific case, set it accordingly. Okay, so let's go back to our terminal again. And now if in order to start the server, I'm going to type in Python, Server, PI, Dash Dash, Threads and 8. That will give us the maximum performance and we simply run this command. Now, after running that command, so it will give me this localhost web address that you can use to actually access the web server. Now you will notice that I'm getting a warning regarding the bits and bytes package because that's specifically designed to run on Nvidia GPUs. So I'm going to simply ignore this. One more thing to keep in mind, you can actually create a public link so you can access your web server that I'm about to show you over the Internet as well. I will show you that in a later video. So what we're going to do is we'll simply copy this and actually head over to our browser and paste it there. And here's the interface that you're going to see. Now, if you have seen my previous videos, this looks very different because they have updated the web interface. However, this is now pretty neat. A couple of things that I really like about it is now in this input box you can define the custom prompt template that you need for different models and we are going to look at a couple of examples in a little bit. But before that we need to actually download a model. So I would recommend to go to this model tab. Right now there are three different ways in which you can download models. This is the one that I really like because it's the easiest one to do. So let me show you how you can actually download a new model. So we're going to move to Hugging face and here you can actually search for different models that you want. So for example, if we go here, all you need is the link of this model. But I'm interested only in this Vicunia 13B, the version 3 model. This is the new model which was released just a few days ago. So I will just copy this link, go back to the text generation web ui, paste the the link here and then simply click on download. Now if we go back to our terminal you will see the download progress is being shown in here. So while this is downloading I will like to show you some other options. First and foremost is the model loader. So most of the models that you are going to be seeing on Hugging Face they are based on transformers. However there are some other models as well. For example Llama CPP based models and there are even quantized versions. So if you're using Auto GPT Q for quantized version of the models. So simply select this. Now here are some more model specific options. So for example you can use a CPU to run them. If you download a full model the way I'm doing right now, you can load it in 8 bits or 16 bit version. Or if you want to unload a part of it to the GPU and part of it to the cpu you can select Auto Devices. There is also an option to load it in four bits and so on and so forth. And you can also use this Trust remote code that will be required for some of the Hugging face models. Okay, let's look at a couple of other things. So if you go to this parameter tab, the most important parameter is temperature. So you want depending on your applications you probably want to keep it to a low value. If you do not want any hallucination in your output. There's this top P top k also repetition penalty. These are different options that you probably have seen. Okay, another thing that you can do is you can actually fine tune your models within the Uber Booker text generation web ui. So I will be creating a few videos on this, but you can essentially Take your own data set and just fine tune models with it in terms of the interface. So if you go to the text generation interface, it looks like this. So you provide input here, it will generate outputs and this works for Instruct fine tune models. However, if you want to have something like Chat interface, you can select this, then click Apply and restart the interface and that will be converted into a chat interface where you can essentially chat with your assistant or your model. So I'm not going to do that right now because we are currently in the process of downloading model, but that's an option that I'll show you later. Now another very important option is actually the prompt template that you're using. So if you go down here, there is prompt and then you have a whole bunch of different options listed in here. So for example, we were going to be working with instructor Kunia 1.3, but it has a very similar prompt template to 1.1. So if you select that, okay, so the installation is done and you're going to see this message done. Now if you come here and re upload all the model's names, so you are going to see a new model. So I'm going to select this and this will load the model. Okay, so the model is now successfully loaded. By default the interface is set to Question answer and that's what the Instruct Fine tune models are. However, you can change this to Chat interface as well. In that case, simply select Chat Apply and restart the interface. Now you will be able to interact with the model as if you are chatting with it. Use this for models that are fine tuned for chat. Now let me show you one last thing. So let's go to the prompt and here I'm going to select instruct Vicunia 1.1. This is the same prompt template that the Vicunia 1.3 version is using. So you see it actually change the prompt template for us if I select another model. So it will use a different prompt template. So it's a pretty nice feature that they have included that you automatically get the prompt template that the model is supposed to be using. Now another parameter that you want to set is the max new tokens. This is the number of tokens that the model is supposed to generate. So I'm going to keep it to to the default 200. But if you want a longer response, you can actually increase this. Now to end the video, I'm going to be testing this model with this very simple prompt. So let's see what it can come up with. So I'll click Generate and this is the actual speed of generation that I'm getting on my M2 machine. I was actually expecting a lot from this model. However, here is what it came up with. Hey there viewer of the video. We hope you enjoyed our latest video about the topic of the topic of the topic and it goes into a loop. But I hope you enjoyed the video. If you did, consider liking the video, and if you are not subscribed yet, consider subscribing to the channel. I'll be creating a lot more videos using this new Webuga web interface. Thanks for watching and see you in the next one.