NEW GUANACO LLM with QLoRA: As GOOD as ChatGPT!

That's how you train a 20 billion parameter model with 40 gigabyte size on consumer GPU with 15 gigabytes of RAM in under three minutes. So they're claiming their models can beat ChatGPT. But more importantly, if their claims hold true, you will not only be able to run a model like this on an iPhone 12, but actually fine tune it, which is crazy. In today's video we're going to be looking at this new paper called Qlorate Efficient Fine tuning of quantized LLMs. So this paper makes some really bold claims. We will look at the output from these models. I will also show you that using that approach, how you can fine tune an LLM on your own data set without the need for a really powerful gpu. So this is going to be a very interesting video. First, let's understand the significance of this work. Most of the models that you see today, they're fine tuned using 16 bit precision, which requires a lot of RAM in order to fine tune them. Now, when it comes to inference, so a lot of people take these fine tuned models and quantize them into four bits. So basically you are taking an already fine tuned model in 16 bits and then compressing it down to 4 bits. Although you will be able to run this on a consumer gpu, but the performance is not going to be that great. However, this paper introduces a special four bit normal float data type that is efficient at being precise. Which means that this technique enables you to fine tune models without losing performance. This process enables you to fine tune a 33 billion parameter model using a single 24 gigabytes GPU or a 65 billion parameter model on a 46 gigabytes GPU. Just to understand how significant this is, if you were to fine tune a model using 16 bits, so you will need 780 gigabytes of RAM in comparison, now you need just 48 gigabytes of RAM. And that is why in most of the papers you see that they're using multiple A100 GPUs. But with this technique, you will be able to fine tune models on commercial GPUs. I hope this gives you a context of why this paper is so important and why everybody is so excited about it. So for the rest of the video we will quickly glance through the paper itself and look at some of the results and comparisons they have done. And later in the video I will show you how you can fine tune a large language model using using their approach on your data set. So they have provided a family of different size Models which they are calling Guanaco. I'm not sure what's with the animal names and large language models, but according to their results, it's able to outperform almost all of the open source models out there. And it reaches around 99.3% of the performance level of Chat GPT while requiring only 24 hours of fine tuning on a single GPU. Now this was done on a single benchmark data set, so these results may not hold true in the world, but I think that's not the significance of this paper. In this paper they are making three different contributions. First, they are introducing a new four bit data type which we already looked at. The second is they quantize the quantization constant. So think about it, they're compressing the compression algorithm and they have optimized the memory load required for loading these models. And one thing that they highlight is that a small, high quality data set can actually produce state of the art results. And this is something that I think the research community has realized recently that you don't need huge models, but you really need to focus on the quality of the training data. So they have introduced multiple models ranging from 7 billion parameters to 65 billion parameters. All of them are trained on open Assistant data set, so, which is relatively small data set. So one of the most important results in this paper is this table 7. So here they compare the responses generated by different models relative to each other on two different data sets. One is the Vicuna benchmark data set and the other one is Open assistant data set. In this case there were two different techniques that they employed in order to evaluate the performance. So they used something called ELIS score. But in the first case, human raters evaluated different responses from the same prompt by different models and ranked them in terms of their preference. In the second case, GPT4 was used to do the same and they're comparing outputs from eight different models, including that from GPT4. And then there's the Guanaco 16 billion parameter model Bard and the ChatGPT. Now before looking at the comparison of Guanaco models with the other ones, one very interesting thing that I noticed was that GPT4 model seems to like its own responses a lot, which is understandable. But the most interesting thing is that even though CHAT GPT is supposed to generate very similar responses to GPT4, but on average GPT4 doesn't seem to like responses from ChatGPT, especially on this Vicunia benchmark. So in this case, based on the ranking ChatGPT is third. Guanaco 16B is second on the rank. But what's surprising to me is that even humans seems to do not like ChatGPT's responses compared to the other models, which is very interesting. However, keep in mind that this, these are the results on a single data set. It may not hold true in the wild. Okay, so before looking at output from the model as well as how to fine tune it, let's look at some of the broader impacts this model can have. So using the techniques, you will be able to fine tune a 33 billion parameter model on a single consumer GPU, such as 3090. And for 65 billion parameter you probably need a single professional GPU, but still not a 100. And to fine tune the 65 billion parameter model, you just need a single professional GPU rather than eight of them. As they note here, even though their models were trained on a very small DataSet compared to ChatGPT, it still rivals it on a benchmark data set. But the most significant part of this work is that using this approach, a small team with small resources can now fine tune their own models for their specific tasks. And one specific example which they talk about is that not only you would be able to get predictions from these models, specifically the 7 billion parameter model in an iPhone 12, but you can actually fine tune the model on an iPhone 12 while it's charging at night. So now everybody can train and fine tune their own personalized LLM on their phones. Now some of the limitations, although they claim that you can get a very similar performance to 16 bit, but it's not going to be the same level because fine tuning a full model on 16 with 16 bit is going to be pretty resource and cost intensive task. So they did not really explore that. And another thing that they highlight in their own work, which is I think very important to do that, they say it appears the performance of these benchmarks likely depend on how similar the fine tuning data set is to the benchmark data set, which is expected. And it's a reasonable issue to have. And that is why I think like we shouldn't be really paying too much attention when they claim that it's almost 99% of the performance of ChatGPT, because this is specifically on this single training data set and it is very important to highlight that part. Okay, let's look at how you can run this or play around with it. So they have provided a playground on hugging face. But more importantly, later on in the video I will show you how you can fine tune an LLM on your own data set. Okay, so here's the demo that they are currently running. It is a 33 billion parameter model, so it's one of the Guanaco model. Okay, so let's look at just a couple of examples. I will be doing a more comprehensive comparison of this with ChatGPT in a later video, but here I just wanted to see what's the speed of generation and everything. So it's an example prompt that they have provided. A llama entered in my garden. What should I do? So let's see. Essentially the response is pretty fast. So if you are unfamiliar with llamas or have never interacted with one before, it is best to proceed with caution. The llama can be quite territorial and protective of their spaces, so approaching them too quickly may cause them to become a crisis. Right. And the response looks pretty good. Now a couple of other tests asked to try to let you Congress in fear of moving to parliamentary form of government rather than presidential form of government. And it does make some good arguments in the favor of parliamentary form of government. So it's pretty nice to see that again. Next, I asked it to generate three startup ideas in the enterprise B2B SaaS, right? And it came up with three pretty helpful ideas. These are not bad at all. Okay, next, I asked it a programming question. So I said write a python function to write a file into an S3 bucket using Boto library, right? And it came up with the code, which is pretty nice. It's a general code. Then what I did was actually I replaced this reading with writing, right? But what I did here is I said this python function is supposed to write a file into an S3 bucket, right? And I intentionally introduced an error. And then I said there's an error in the code, can you highlight it? And it's actually able to do that. For example, your open function has an incorrect parameters. To open a file in the Text Format Pass RT2 mode and to open a file in the binary format pass RB to the mode. But I was passing WB in the incorrect code. So this is actually really impressive that it can figure out what was the problem. I would say I'm actually impressed when it comes to its coding ability. Okay, one more programming test. So I asked it to write an HTML code for a website that has a single button. When the button is pressed, it randomly shows one joke from a list of 10 jokes and it will also has to change the color. Now, it wasn't able to complete this code because I think it exceeded the token video so what I did was I repeated the same prompt, but this time I said three jokes. Right. And here is the code that it came up with. Now let's copy this. I'm going to go here, paste it here and let's see if it actually works. This is going to be impressive if it does work. Oh, there is a button. Although probably I should have asked it to make it bigger. But let's see, it changed the color and it also showed one of the jokes and oh wow, okay, it works. And this is crazy because imagine you can potentially run this on your phone or on a local machine. And this is really impressive for me, especially for a programmer. In most of the other open source large language models that I have tested, I think only wizard Vukunio was able to do this task. So I am impressed. But we're going to do a more thorough comparison between this and other models in a subsequent video. Okay. Okay. Now so we looked at the capabilities of the model, but let's see how efficient this is. So they have provided two different Google Colab notebooks. One of them is to show how to use a 4 bit model in inference with all their variants. And this is a model which is a 20 billion parameter model. So usually you are able to run a 7 billion parameter model with 4 bit quantization on a free Google Colab instance, but they are able to run a 20 billion parameter model, which is impressive. And the second notebook shows how you can fine tune a model using four bit quantization that they have proposed. So we're going to look at an example of how to quantize the same 20 billion parameter model on our data set using just a Google Colab. So Google Collab has a T4 GPU which is decent, but not great. Okay, so let's first look at the inference notebook. Now in this notebook they're using this bits and bytes package which integrates the approaches presented in their Qlora pip. Now in the first section they are showing how you can load any model in four bit. And this is going to work for any model that supports Device map. So here is a basic example. If you want to load a 30 million opt model, right? So you can do that using the auto model for Causal LM. And similar to when you load a model in 8 bits, you can now pass on load in 4 bit equal to true. And this will load the model in four bit. And once loaded you can get the predictions in exactly the same way you would do it using the transformer library. I would recommend everybody to Go over the rest of the sections in this Google Colab. But the thing that I want to look at is this one program pushing the limits of Google Colab. So in this case they're actually loading a 20 billion parameter model. So this is a GPTA Neo X from Luther AI. Again they're using the bits and bytes package and loading the model in four bit. But now the quantization type is the new float type that they have introduced in their paper. Now the rest of the process is exactly the same that you would perform in transformer package. You can actually get predictions using this 20 billion parameter model, which is crazy because this T5 T4 GPU that Google provides, it has only 15 gigabytes of RAM. Okay, next we will look at this second notebook which shows you how you can fine tune a model based on your own data set. And I would actually encourage everybody to experiment with this. The data set that they are going to be using in this example is this English quotes data set which is available on Hugging Face. So it contains a JSON file in the following format. So it has a key for author, then a given English quote and then some tags associated with the quote. So while the Google Colab is is running. Let me walk you through the code. So first we install all the required packages. So this includes the bits and bytes, the transformer, the perf, that's the the algorithm used for training Accelerate and then the data set package. Next here they're configuring base model. So again this Notebook uses the 20 billion parameter model up from Eleuther AI called GPT New X20B. And this is a pretty huge model. The size and half precision is around 40 gigabytes. But the T4 GPU that we are going to be using in this, it has a memory of only 15 gigabytes. So again in the configuration we set it to be in the 4 bit and we select the special 4 bit data type that is presented in the Qlora paper. So this is the normal setup that you need to provide. Next we will have a tokenizer that is coming from the transformer library. And the tokenizer is going to be based on the model that we have selected. And you simply set up the model the way would you would do it using the auto model for causal inference. While this is downloading, let's look at the next part. So once the download is complete, Next we are just setting up and some configurations for training the model. Now since we're going to be using Lora for training, so it's only using 0.01% parameters for training. So small fraction of them are going to be trained, and those are part of the LORA adopters. Next, we need to set up our training data set. So in this case they're using the English quotes data set that I just showed you. So here's the format again. So quotes, then the authors and the corresponding tags. And we just want to look at the process data set. So in this case what they did was they tokenized just the quotes. Right? So now you're going to have a couple of other fields. One is the input IDs, and this is based on the tokenization process and then attention mask as well. These are the things that are added by the tokenization process. Okay, one more thing. There are just 2,500 examples in this small data set. So it's not a large data set. But just to show you, you can actually fine tune your model. Next is the training or fine tuning part. So they're setting the maximum number of steps to 10. If you're doing it on a large data set, you definitely want to keep it much higher. But essentially using the transformer package, we are going to be retraining the model. So we pass in the model, the corresponding data set that we just created and some other training parameters. For example, here's the learning rate. Then floating part precision that we want to use is 16 bit, right? And some other things. So let's run this and see how long the training process is going to take. Okay, so the training process is complete. So that's how you train a 20 billion parameter model with 40 gigabyte size on a consumer GPU with 15 gigabytes of RAM in under three minutes. Now we'll definitely have to train it for a lot longer for the training last to start decreasing. But this is just the process of how you can fine tune a model using this new Qlora approach. Okay, so you're able to train the model, but how do you get predictions from it? So it's an autores regressive text completion model. So we pass on our text, right? Then pass it on to a tokenizer and then pass the input, which is tokenized version of our input into the model and call the generate function on it, right? And then we simply decode it. So here was my prompt. Hello, my name is. And here's the response that came up with. So, hello, my name is John and I'm a programmer, I'm a pro because I like to write code. You can also integrate this as a pipeline from the Transform package. If you want to get predictions it's just fascinating to see the progress that has been made by the open source community in the last few months, especially since February when the Llama Virtual released. I can't wait to see what's going to happen in the next couple of months. Just quoting the words of a wise man, what a time to be alive. If you liked the video and found this useful, consider subscribing to the channel. If you haven't, and push that Bell notification button to get notifications for each and every new video. If you were working on a custom large language model or information retrieval models and you need help, you can actually consult me. So there's going to be a calendar link in the description of the video. We also have a growing Discord community, so join the Discord if you are interested. I do have some good plans for the community over there in the coming few weeks, and if you would like to support the work that I'm doing, check out my Patreon. As always, thanks for watching and see you in the next one.