LLAMA-3.1 🦙: EASIET WAY To FINE-TUNE ON YOUR DATA 🙌

This plot shows that first time in history the open weight models are closing the gap in terms of their capabilities to the closed source models. We are able to do a lot more with less number of parameters. For example, the current Llama 3.18 billion model is very close to the previous generation Llama 70 billion model. This shows smaller models are better candidates for fine tuning today. And that is the goal of this video. I'm going to show you how to fine tune your own llama 3.1 using unslot. Why Unslot? Because it gives you faster training at much lower VRAM requirements. It's critical if you are GPU poor like me. They also release their own chat UI for chatting with LLMs, which we're going to cover later in the video. Before looking at the code, let's talk about different stages of training. Usually there are two or three different training stages. The first one is called pre training, where the input is a bunch of raw text data and the model learns how to predict the next token or next word. The result is a base model where the input is going to be the raw text. The output is going to be the most probable next token. Now, during this stage it acquires a lot of knowledge from the text. However, it's not very useful. To make it useful, people usually do the second stage, which is called supervised fine tuning. And in this case the input is going to be question answer pairs or instruction and answer pairs and the model is supposed to learn from that data. So in this stage the input is going to be a question or instruction and the output is supposed to be response or answer. Now there is a lot of debate whether during this stage the model can learn any new knowledge or not. I'm going to be addressing that in a subsequent video, so stay tuned for that. We are going to be doing instruction fine tuning in this video and the result is going to be an instruct model. The third stage is preference alignment. If you have heard about RLHF or algorithms like dpo. In this stage we want the model to learn what the user prefers in terms of responses, or we want to align the model to certain set of principles. Now, this is an optional step, but a lot of the API providers include this and the result is going to be a chart model. Let's talk about different supervised fine tuning techniques. Since we are going to be doing supervised fine tuning in this video, there are three popular options. The first one is full fine tuning. In that case, you take the original model and you update the weights with the instruct fine tuned data set that you have now. This will give you the best performance depending on how much data you have in your Instruct fine tune dataset, but the VRAM requirement or the GPU VRAM that you are going to need is going to be pretty high. The second option is to not update the weights directly but add external adopters to it where you can control the number of parameters in these external adapters. This approach is called Lora and usually you do the weight updates for these external adapters in 16 bit precision and later on merge those adapters to the original model weights. We're going to talk about this later in the video. This will give you quick training but will still be costly because you're doing all the operations in 16 bits now. The third option is to use Qlora where you are essentially doing the Lora adopters but the weights Updates are in 4 bits precision rather than 16 bit precision and you also keep your model weights in 4 bits. For this you don't need a lot of VRAM but the performance is not going to be as great as Lora or Full Fine Tune. Unslot supports both Lora as well as Qlora for fine tuning. The images I was showing are coming from this blog post finetune llama 3.1 ultra efficiently with Onslaught I'll put a link to this blog post and I'll highly recommend to check it out. Here's the official notebook from the Onslaught team on how to train the Llama 3.1 family of models. We're going to be using a free Google Colab notebook. That's why I will be fine tuning the 8 billion model, but you can do the same with the 70 billion model as well as 405 billion model. My focus for this video is going to be not only to show you how to do fine tuning, but spend more time on how Lora and Qlora works internally, how you can set up the different hyper parameters in Lora and what is the impact of those hyper parameters on the memory requirement. Okay, so first we need to install all the required packages that will include the unstopped package as well as a number of packages from Hugging Face. That includes Peft. Peft is actually used for for the internal fine tuning. We will also use the Accelerate and Bits and Bytes package. Next we will import the Fast language model class. This is going to be used for loading the tokenizer as well as the model in unslot format. You want to set the max sequence length. The 3.1 llama family supports up to 128,000 tokens. However, if you are fine tuning a model, you need to set this based on your own training data. So look at the examples that has the max sequence length and set it up to that. Keep in mind, if you set it to a very high value, the amount of VRAM that you will need is going to increase substantially. We are going to be doing everything in four bit quantization. Now, Unslot comes with a number of different models. In 4 bit quantized version, they even have a 405B, which is pretty crazy. But in this case, we are going to be loading the model and we'll do the quantization later during the training. So we use the fast language model class that will load both the tokenizer as well as the model. We are setting the max sequence length to 2048 tokens and we're going to load the model in four bits. Next, we're going to set up our Lora adopters. Now, there are different modules within the LLM that you can set up which are going to be changed. But a couple of things that you want to keep in mind is how to set up this R, which is the rank of the model, and what type of impact this is going to have on the number of parameters that you will need. The second one is the LORA alpha and the LORA dropout. Now, to understand the rank, let's look at how LORA actually works. So if you recall, we have these Lora adopters. There are two different matrices. One is A and the other one is B. And there is the original weight matrix of the model, which is represented by W. To make it simpler, I created these three different matrices. So the original matrix is, let's assume thousand by thousand rows and columns. That is a pretty huge matrix. If you're doing fine tuning or weight updates, that will require a lot of vram, especially if you're doing it in 16 bits. But you can decompose this into two different matrices. Now, these are independent of the original weight matrix. So one is 1000 by 8 and the other one is 8 by thousand. If you multiply them, its size is going to be exactly the same as the original weight matrix. So instead of directly training or updating the weights in the original matrix, you can now just update the weights of these two matrices. Now, the number eight represents the rank of your Lora adopter, which basically controls how many parameters are going to be in the final matrix. So you basically take these two smaller matrices, multiply them, the result is going to be the weight update and you just add them back to the frozen weights, which are the original weights of the model that will give you the final weights. And we do this updates on each iteration of the training. Now, the rank, or lora, as I said, controls the number of parameters. So if you keep the rank low, the number of parameters that you are training are pretty small. That means it's not going to have too much impact on the actual weight updates. So you can actually increase the rank, but that means that you're going to be adding more parameters. That will mean more vram. Okay. The second hyper parameter is Lora, which basically controls how much contribution these LORA adopters are going to have on the final weights of the model. To understand this, let's look at this diagram. So here we have a matrix which is basically the result of multiplying those two LoRa adapters or LORA matrices. Now, we need to add this back to the original frozen weights in order to update those weights. But how much that contribution is going to be controlled by our LORA alpha. So if you set this LORA alpha to a higher value, that means stronger adaptation. If you keep it to a lower value, that means weaker adaptation. So this will basically control how much impact these adopters are going to have on the original weight metrics. Now let's talk about data preparation, which is a very important part when you are fine tuning your model. So here's how the prompt template looks like for Dhamma 3.1 family. This is a special prompt template that you will need to use if you were to fine tune the instruct fine tune version. However, since we are fine tuning the base model, we get to choose our own prompt template. So we're going to be using the Alpaca prompt template which comes with an instruction, which is below is an instruction that describes a task paired with an input that provides further context. Write a response that appropriately completes the request. So we're going to provide an instruction input and the model is supposed to generate a response. For fine tuning, we'll be using this Yamaha Alpaca clean data set which has output, input and instruction. Now, not in all cases, you have an input. So the instruction directly is going to become the input to the model and the model is supposed to generate these responses. Also you need to set end of sequence token. If you don't do that, you're going to run into an infinite loop where the model will just keep generating text until it runs out of tokens. And this is, I think one of the issues that a lot of people ran into when they were trying to use the quantized versions using NAMA cpp. So we load the data. If you have your own data, you will need to just set it up like this. So basically we take all these three columns and convert it into this specific format. Now in order to train the model, we're going to be using the SFT trainer from TRL package which is created and maintained by hugging face. So we provide our model tokenizer, the data set. In this case we are going to say that we want to use the specific field called text. Because when we set up our prompt template or reformat our prompt template, we are using this as a text column which is added back to our original data set. We set the max number sequent and then some arguments for our training run. So the T4 GPU which is freely available on Google Colab has about 15 gigabytes of VRAM. It's going to be using only 6 gigabytes out of that, which is pretty awesome. Now next we just run the trainer on it and you can see the loss is decreasing over time, which is a pretty good indication. Now here are the numbers when it comes to the training. So it took about eight minutes to train. Peak reserved memory was about 8 gigabytes, which was about 53% of the peak memory that is available. So not bad at all. Now to do inference we can just use the for inference class or of method on the fast language model class. So we provide our train model and then we need to provide the input or the prompt in the ALPAVA alpaca format. So we use the same function to convert our prompt into alpaca. Then we set the max number of tokens that we want the model to generate and we can display the tokenizer as well. So here is how it's going to look like. And here's the response from the model. With this you can also enable streaming, so you create a streamer object and just pass that on. So now the responses that you're going to see are going to be streamed one token at a time. The next section is talking about saving loading fine tuned models. So you can call the save pre trained function on the model to save it and then you can use the same function on the tokenizer to save the tokenizer as well. So the result is going to be a number of different JSON files which will contain the model weights as well as the tokenizer. And in order to use that you just use the same file language model class from pre trained and instead of providing the repo id, now you just provide the local directory where you stored the model and you can run the same thing as before for prediction. So pretty neat. Another great feature of Unslot is that you can save these models to different formats. So for example, you can save it to 16 bit floating point precision for vllm, or you can use or you can save this directly in GGUF format for llama cpp, which is pretty neat. And as I said in the beginning of the video, they also introduce a new UI where you can run the models that are trained using onslop and you can chain chat with the model. So this is based on radio, so I'll show you a quick demo. Here's a Google Colab notebook that they have provided so you can directly clone the repo, which is the studio repo, and then run that code directly here. Okay, so you'll have to wait a little bit, but once it's done you're going to see a UI like this and now you can start chatting with the model. So the first prompt that I'm going to be using is this. Which number is greater? 9.11 or 9.9? Which got pretty famous. And in a subsequent video I'll explain why this is a terrible prompt to test an ll and you can see the LLM is not able to get this in this case I think we're using the 8bit llama 3.1 model. But in general you shouldn't be using the prompts like these to test these LLMs. Anyways, I'll explain the reasoning why most of the LLMs gets this wrong in another video. So I hope this gives you a very good overview of different stages of training. How do you do Lora fine tuning or Qlora fine tuning with unstopped? What are the different hyperparameters of Qlora and how they impact the size as well as the performance of your final model. I hope you found this video useful. Thanks for watching and as always, see you in the next one.