GPT4ALL: EASIEST Local Install and Fine-tunning of "ChatGPT" like MODEL

There's a new open source GPT model in town and it's probably one of the best and easiest to install on your local machine. It's called GPT4ALL and is available on GitHub. In this repo, not only you can use this model locally, but they open source the data set. So you can potentially train and fine tune this model as well. It's based on llama 7B and it's probably one of the easiest to install compared to all the other models. So in this video I'm going to walk you through a step by step process of how to install it locally, run it and potentially how you can retrain this model on your own data set. But before the installation, let's look at the technical report to understand what is going on in the model. So the technical report is titled GPT for all Training and Assistant Style Chatbot with Large Scale data distillation from GPT 3.5 Turbo. So it's actually trained on the data from ChatGPT. Now in the abstract they say that this describes the development of GPT for all a chatbot trained over a massive curated corpus of assistant interaction, including word problems, story descriptions, multi turn dialogue and code. And they have made available the data set data curation procedures. So if you want to create your own data set that is included, the processes included, training code and final model weights are also included. And they have released a four bit version of the model and you can virtually run it on a CPU. Now they use ChatGPT and they collected over 1 million prompt response pairs from ChatGPT in a span of a week or so. So those are the dates and then they did some filtering process. So overall their total number of example was around 800,000 high quality prompt generation pairs from ChatGPT. And then they further distilled it into I think around 400,000 prompt generation pairs. So this is actually the data set that they used for the training. Their final model, if you remember Alpaca was fine tuned on only 52,000 prompt generation pairs. So it's almost 10 times, 10 times more data in this case. Now in terms of the training process, so they fine tuned from an instance of Llama 7B and it is trained with Lora, 400,000 examples. So let's look at the training data set and it's actually pretty nice. They have provide data cloud. So here you can actually see for example, and these are different topics on which it was trained, right? And it's a pretty nice one. So let me zoom in and you can pick one example like let's say President of the United States, right? So if you click on any of the data points then you will see the ID number, the prompt that was provided. What was the response from ChatGPT? So these are the red example and this is exactly the way you need to structure your data set. If you are fine tuning this model. So it's supposed to be a JSON file with this structure. So you can actually look at different examples and see what type of data was used. But this is overall a pretty nice way of having more transparency. All right, next we are going to look at how do you actually download and use this model. So as I said before, it's one of the easiest process I have seen. And before going into the training part or fine tuning, let's look at it how you can locally run this. Now in order to locally run this, come to this code, click on it and then if you know how to use GitHub, you can actually clone this repo. If you don't know GitHub, that's absolutely fine. All you need to do is come and download zip. So it will download download the zip file. I have already downloaded a zip file. So whenever you download it, just extract it and you will see this GPT4 main GPT4 all dash main. Right? And then you will see a few files. So let me actually make it large, I think that will help. Right, so you have a few files and then a couple of folders. Okay, so that's step one. Now step two, you need to actually download a model checkpoint. So they have provided this link and you will download that a file. It's around, I think 4 gig file, so it's going to take a while. I already have downloaded it, so I'm going to be using that. But once you download it then you need to go and paste that file in this chat folder. Okay. If you see in my chat folder I pasted this GPT for all the LoRa quantized bin. It's around 3.92 gig, so around 404 gigabyte file. Okay. All right, next, if you are on Windows or on Mac, they have the given the instructions. So simply open the terminal, right? Change directory to the chat folder where you just copied the file and then you need to run this file. Now I had a small issue on Windows. Let me show you. All right. So I went to the main folder and then I'm trying to execute the command that they have mentioned here on the Windows platform, right? But I ran into this problem that the system cannot find that path. But if I go here, right? And try to execute it like this. So the cd what it does it, it will change the directory to chat, right? And then it tried to run this file. Let's see, I still get an error. Let me manually do this chat, right? But if you say dir, there is a file, this file, right? And now in order to run it might actually. But I'm going to show you how to actually run this even on the Windows platform. But you see, dot is not recognized as an internal or external command, right? That is okay. Okay, so it's looking for a program to open this dot BIN file with. Okay, so no issues at all. Let's fix it. If you are on Windows 11, right? Then you need to. So go to the Start menu, right? And look for wsl. Wsl. All right, so you will see something like this. If you don't see this Windows subsystem for Linux, you need to install it. So in order to install it, go to your Microsoft Store and in the search simply type in wsl, this Windows subsystem for Linux, right? And I have it already installed. But if you need to install it, just click Install it will install it. It's a very easy process. You need to set up your user account, but it's a easy process to do. So I already have it, so I'm going to open that instead. All right? Okay, so this is the folder where we are in right now. So I'm going to say cd, then CD again. All right, so if I say ls, this gives me a list of all the folders that are available in C. So I want to change directory to this path. The folder that I downloaded is on my desktop, so I actually put it here. And here's the path relative to the C drive. So let's do that. Okay. And LS LS means list all the files and folders. Okay, so now we have the ones that are highlighted. These are actually folders. So now CD into chat, okay. And LS again. All right, now we can actually run this file. All right, so to run this, I think all I need to do is I can type in GPT T for quantized bin, right? Oh, no, not that one. Quantize Linux. Actually, I want to run this file. Okay, so not the BIN file. All right, I think we're all set. Let's run this guy. Okay, so one, this is loading, right? So it's actually loading the model that we copied in there, right? While it's loading, let me show you the resources it's utilizing. So right now my memory is around 90%. I'm running this on a laptop which has a memory of 16 gig, right. So it's going to take quite a while, quite a lot of memory. So just keep that in mind while you're running this. Okay, so it has loaded. That's a good news. Right, now let's see how good it is. Right. And the way I like to do these experiments is I want to first check it on exactly the commands that they used. Right. And then we could do our own experiments. Okay. And remember, you can actually chat with it. Right. So they have provided a few examples. Let me see. All right, so for example, provide instructions for a given exercise. Select races. I'm going to copy this, right? And let's paste it here, run it and it will start processing. Just keep in mind that it's doing all the processing on cpu, so it's a bit slower. If you have a gpu, then it will be a bit faster. Okay, so it's coming up with the instructions to perform leg raises. You will need to lay on the back with both legs extended. All right, so it's actually provided, I think, a pretty good instructions. Let's ask it recommendation some leg exercises. So the exercises are great way to strengthen and tone your legs. There are many different types of leg exercises. Okay, so in terms of the exercise is actually it's pretty accurate and it came up with a exercise plan as well. Let's see how many times a week a week we need to do this. Right. It's actually pretty nice. So it depends on what you are trying to achieve with these exercises and how much effort time you have. All right, so generally it is recommended that people perform strength training two to three times per week. Okay. Like overall, it's actually pretty coherent and it makes a lot of sense whatever it's saying. But these are simple questions. So let's. For me, let's see if you can ask it to write a program. Okay, so my instructions were write a pro Python program that expects an input n from the user and then prints the corresponding Fibonacci sequence. And it actually started defining a Python function. So if not instance n. Okay, so it is expecting an integer, checking if it's an integer or not. Right. If it's not an integer, then it will simply return invalid input. It didn't complete the function. Okay. I think it's hallucinating for programming because it did manage to reverse it, but then it's simply repeating itself and then like it's checking these types in there. I don't know what it's doing. So I would say the programming is probably not great, but let's check out the other capabilities. All right, so I asked him again another python programming question and this is the one that actually they mentioned here in example seems to have implemented correctly. I think it sits right can do programming, but I don't, I won't rely too much on it. All right, so I gave him a text is actually a point that it generated and it's an example from the website and actually it did a really good job just job at summarizing this. I'm kind of impressed actually. All right, so a couple of other questions. I think the mandatory what is the difference between alpaca and nama? And it seems to have the difference. And then I asked it come up with an interesting idea for a new movie plot. Your plot should be described with a title and a summary. So Alice in the World in Land is the story of Alice who falls down into a rabbit hole where she meets various characters. So it seems to be coming up with something. Okay, it came up with a plot. So it seems to be creative for sure. And even the execution time is pretty good. So it's on par with ChatGPT, I would say like in terms of the speed of coming up with the tokens. Right, so now next let's look at how can you potentially train it on your own data. Okay, so to look at training or fine tuning. So in their technical report they have this section on reproducibility. So we release all data, training code and model weights for the community to build upon. Please check the GitHub reposition repo right up to the data, training data and everything. Okay, now here is the most important thing which you need to consider. So we were able to produce these models with about four days work in total. Right? So 800 for GPU cost and then 500 in OpenAI has spent. So this is basically when they were creating the training Data set using ChatGPT. Okay. And can be trained. So the model can be trained about in about 8 hours and you need DGX A100 GPUs to actually train it and the total cost is going to be around hundred dollars. Now it doesn't have to be this because if you're training it or not smaller data set, it's going to be lower than this. Okay, so so far we talked about running the models on your local machine. Now let's talk about reproducibility and potentially Retraining the model. So everything you need for reproducibility is actually provided over here you have access to the training that data that they used. So I would recommend using this second data set. And then they actually even provided the weights that you need. So now you will need to copy these as well. Now, in order to set up, all you have to do is clone the repo, right? And then configure using this command. Then you need to set up a Python environment and this command line will simply install all the required packages. You will need to do the next couple of steps as well. So all the dependencies will be installed after this test. This steps. Now, in order to retrain the model, you need to run this command. Let's say if you are using an EC2 instance or even on your local machine, if you have powerful enough gpu. So you need to run this. But there are a few important things that we need to look at. So for example, here they're running this data set. Configure JSON file train py, that's the Python training file that it will run. And then there's a configuration file, a YAML file. So let's look at it. Where you are going to provide the data, where it's going to read it, where you want to store your file, the store your final model as well. All right, so if you come up to the main repo, there are a few things where you need to look at. So first I'm going to open this data PY file, okay? And here if you simply look for a JSON, right? So that's the part of the JSON file that has the training data. And this path is coming from the YAML file here under the configs. So config train. And then depending on whether you are fine tuning the LORA or the normal fine tune, right, so here's the data set underscore path. So you're going to define your path here. And so basically it's going to be a folder where you're going to put your training data in JSON format, the resultant model that is going to be stored in this file structure. So you can actually change the path if you want. Sorry, this is going to be the file name, but the path is going to be here, okay? That's the output path. So now the training data is available. If you want to provide your own data, you are going to have to structure it in a JSON file. Like in a similar fashion, for example, you need to have an ID prompt and then a response. So that's going to be the structure. You can learn more about the structure of your file by looking at the documentation of the Llama and Alpaca models. But keep in mind that you will need a very powerful machine to actually run the training. So for this, almost 400,000 examples, it took around eight hours on a Lambda Labs, a 100 GPUs right. So you could potentially run it locally, but it might take a lot of a long time. I have a machine coming up, so I have a 3 GPUs machine that I ordered. Once it arrives, I'm going to definitely try training on it. But right now I'm looking at some cloud platforms to run this training. And once you train your model in terms of generation, you can use Python to actually get a response. So this is the format you're going to be using. You're going to provide the configuration file, this is the inference Python file that you need to provide. And the prompt, whatever you want to, whatever the query is, you simply provide that. All right, so I think that's it for this video. I hope this was useful and you learned something today. It's a really nice package. Give it a shot. It was just released, it was announced actually today. So probably you're going to be seeing one of the first few videos that are made on this package. So let me know what you can come up with. If you have any questions, please don't forget to comment and like the video. Thanks for watching. See you in the next one.