Get your own custom Phi-3-mini for your use cases

Phi 3 Mini is an amazing model for its size, but you know what's better than this? Phi 3 mini fine tune on your own data set. So Microsoft released the weights of the model and with this it also released an example code to fine tune the model on Multi GPU system using the TRL and Accelerate modules from Hugging Face. Now this uses the Transformer package which is relatively slow when it comes to fine tuning LLMs. But the good news is that the Onslaught team recently integrated the ability to fine tune 5, 3 models into the Unslot package. Now with that you are going to be able to fine tune models much faster with less memory usage and increased speed of fine tuning. And that's exactly what we are going to be using in this video. So we will be using the official notebook released by the Unslot team. I made a couple of adjustments and I'll walk you through those. So first you need to install all the required packages. These two commands will download and install both Unstloth xformer X rate Bits and bytes and PEFT libraries. Now if you have seen my previous video on Fine Tuning Lemma 3, the steps are pretty much exactly the same. In this case we are importing the fast language model class from Unslot. This is the main class that will load the model and also takes care of adding the lower adopters as well as the fine tuning. Now you need to set the max sequence length. This is basically the maximum sequence length your inputs and outputs can have. Now it's set to 2048 tokens, but since Unslot supports rope scaling, so you can basically choose any value you want. Currently we are going to be fine tuning the 5.3Mini 4K instruct version, but there is also 128,000 token context render version as well. Now here's the list of all the models that are currently supported in Unslot and they actually have a four bit quantized version of those that are already stored in their Hugging face repo. So we are going to be using one of their version, but you can pick any model that is available on Hugging Face and use it with onsloth. So we are using the Fast language model dot from pre train. Then we provide the model name the max sequence length that we set here. Also we need to provide the data type. So data type. If you set it to none depending on your hardware or GPU it will automatically select the most appropriate one and we want to load the model in four bits. Now if you are using a gated model like llama 38 billion from Meta AI. Then you will need to provide your hugging face token as well. Okay, so once you run this, this will basically download the model in your local environment. Okay, now this is the base model, so we need to actually add Lora adapters on top of it to do efficient fine tuning. Lora adapters are additional layers that you use to update during this fine tuning process rather than touching the actual weights of the model. So here we are actually adding these additional layers along with some hyper parameters that we are setting to the model. And that way we're going to be just fine tuning or training the Lora adapters, and later on we will merge those Lora adopters to the model. Now, if you have seen my previous video, I go into a lot more details. I'll put a link to a couple of videos that talks about Lora adopters and how they work. All right, the most important part is the data preparation. And with the like, if you understand this section, then you will be able to structure your own data sets in proper format. Now, in this case, we are going to be using the alpaca prompt. So this has three sections. One is instruction that are specific instructions to the model, what to do, then input from the user and then the response the model is supposed to generate. Right? So your training data needs to have these three different parts, or at least the instruction and response part if you're using the alpaca format. Now, in this case, we are going to be using this data set, which is a clean version of the original alpaca data set. The alpaca data set was created with the help of GPT4. So it has three parts. One is the instruction, this as a system instruction. Then the corresponding user input, like you may or may not have user input, right? So in case if user input is missing, it will simply use these instructions to generate a response. And then at the end you have the output, which is what you expect the model to generate. Okay, so now if your data set has these three columns, then we can use this to actually format the data for us. So let me walk you through how the data formatting is done. So we load this data set and we are just loading the train split. Then we are calling this function on each and every example, and for each example or sample, it isolates the instruction input output and then it applies this prompt template to each of these different sections and put everything together in a new column called text. All right, so after that processing, here is how our data set looks like. So instead of the initial three columns. Now it has four columns and there are about 52,000 rows. Now we randomly picked a couple of examples. So I looked at the couple, the first example in the data set and here's how it looks like. So here's the main instruction of what the model is supposed to be doing. The instructions from the user here is there's no input for this specific example and this is the output generated by the model. Now, in this case we have an instruction which is write a short paragraph about a given topic. Then the topic is the importance of using renewable energy. And here's the response that we want the model to generate, right? So make sure to structure your your data in this format if you want to use the Alpaca A prompt with fine tuning the Phi 3 mini model. Now another important thing to consider is that you also need to add end of sequence token. This is important because if you don't add that, the model will just keep generating forever and it's not going to stop when it's finished generating a coherent response, right? So this is very important that you actually need to add this to your text column that you create and it's going to be added to each and every example. All right? So that was the data formatting part, which is the most important part. So you need to make sure that you understand that correctly and you structure your data correctly. Now the next step is to start setting up our safety trainer. So behind the scenes, this is using the TRL package from Hugging Face and we're going to be using the SFT trainer class, which is supervised fine tuning class. Now we need to provide the model that we created using the onsloth package, the corresponding tokenizer. So this will basically tokenize the data depending on the tokenizer that was used to train the model. Then we need to provide our data set and we need to tell it which column to use. So if you recall from above, we set the data in a proper format using this text column. Next, we need to provide the maximum sequence length, right? So the max sequence length is 2048 tokens in our case, right? Okay. So once we set this up, we also need to provide the batch size. Now this is going to be dependent on the GPU memory that you have. You can probably define higher batch sizes, but start with a lower batch size and then gradually increase it to see how much larger batch size your GPU memory can support. All right, now we're going to be running this only for 60 steps because I just want to show you how this works. But if you Want to train a model and want to have better performance out of it. You want to make sure that you run it for. The other important aspect is the learning rate. So this will, this will basically control the rate of convergence. So start with a relatively smaller learning rate, that is fine. And then we are going to define an output directory where it's going to basically store the model and all the corresponding loggings are going to be stored in here. So once we run this, this will set up our trainer for us. Now here's a simple comparison of how much the memory is going to be used. So we're using T4 Tesla, which is the free GPU available on Google Colab, and it has a GPU VRAM of about 15 gigabytes. With this setup, we expected to be using about 0.3 gigabytes of memory for fine tuning the Phi 3V models. In a previous video I actually showed you that you can use the free T4 GPU to fine tune even the llama3.8 billion model. And we had enough memory left even in the 15 gigabytes VRAM. Okay, so once we set this up, we are going to call the train function on our trainer and, and here you can see how the training proceeds. Now we are just providing a trained data set, we're not providing a validation data set. So that's why you only see a training loss. Training loss starts at pretty high value, 1.13 and throughout the training it gradually decreases. So this is an indication that the model is actually learning. Now, in practical use cases, you definitely want to have a separate validation set to also track whether the model is overfitting or not. Right. But since we ran it for only 60 steps, so this is not significant at all. Like I don't expect there is going to be any overfitting. All right, so let's look at some of the stats. So here is the number of seconds that it took. The train samples per second was about 1.75, so not bad at all. And we just ran this training cycle for a fraction of the epoch. Right. So if you want to run it longer, you can actually set the effect to one or at least the higher number of steps. Now how did the training look like? Let's look at the memory. And time starts. So it took about five minutes for 60 steps. Right. The peak memory reserved was about 2.8 and the peak reserved memory for training was actually just about a half a gigabyte. So it's a pretty impressive because it was using only about 18% of the max Memory that is available. So this is pretty impressive. And you can see the unslot is able to manage memory very efficiently. All right, so let's say we have trained the model. Now how do you do inference? So we're going to again load our model from the fast language model, but now we're setting it for inference and we need to provide our input. So the input again needs to be structured in the alpaca format. So here's the instruction. Then we have our user input, and then the model is supposed to generate output. Okay? So we provide these three different values and wrap it around the alpaca format. So that is going to become instruction input and it is expecting an output. And we feed that into the tokenizer too, create a tokenized version of our input prompt. And we're going to run this on the available gpu, right? And then we pass it on to the generate function on model that will generate the response. Now here we are restricting the max number of tokens that the model is going to generate to 64. Just play around with it. If you want a larger response, you want to set this to a higher value. So here's what the output looks like. This is basically the initial system prompt, then instruction from the user, which is continue the Fibonacci sequence. This is the input. And here is what the model generated and which is pretty neat. Now, unsolved also supports streaming the response that it generated. So here we set the Max tokens to 128, but now we are using the text streamer. So if you run this, you are going to see a streaming response for the same prompt. Now this was how do you get inference from the model that we just trained? But you definitely want to store the model and you want to load it for future usage. So in order to do that, we're going to be calling the save pre trained model, then provide the model name. This will basically store the model in loramodel folder and these are going to be the content of the model. You can also push this to the hub. In that case you will need to provide your username and then the repo that you want to store it in and the corresponding hugging face token. This will push the model to the hub and then you can download it from the hub and use it exactly the way we have been doing it. Now here's basically loading the model from the local storage that we just stored, right? If you don't want to load that, just set this to false. But I set it to true. And we are again Using the fast language model class, it will load the model, we set the max sequence length, we are loading it in four bit, right? And we need to actually set this for inference so that we do inference on the model and exactly the same way that we had set it up. So you will need to provide your instructions, inputs, wrap it around in the alpaca format, tokenize it, put it on your GPU and then call the generate function along with the max number of tokens that you want the model to generate. So in this case the prompt was what is famous tall tower in Paris? And here is the model response. So it says the famous tall tower in Paris is Eiffel Tower. It is an iconic symbol of France and one of the most visited tourist attraction in the world. Right. And it goes on to add some more details. Now, the notebook also talks about that you can use the auto model for peft causal LM for inference, but they don't really recommend it because it's much slower than Onslot's implementation. Now another great feature offered by Onsloth is that you can save the model in 16 bits for VLLM. VLLM is another inference library that can provide you high throughput if you're doing inference of different models. So here you can define different quantization levels. And the way you do it is you define the save method. So if you say I want to use the 16 bit merge version and 16 bit quantization, just say merge underscore 16 bit. You can do the same thing in 4 bit as well. And you can also just store the Lora adapters as well. Now another amazing feature is that you can directly create the GGUF version of the model which you can use with LLAMA cpp. So that's pretty easy to do. Again, in this case you just define the quantization method. So this is floating point 16 bit. Here is the 4 bit quantized version. I think if you don't define anything, it will automatically store the model in 8 bit quantization. Okay, so this was a quick video on how to fine tune 5.3Mini 4K instructor version on your own data set. The most important part is how you structure your data set. So I'll highly recommend to look at this section on data preparation. I'll probably create some follow up videos in which we are going to create our own data set rather than downloading them from hugging face. But irrespective whether you're downloading them from hugging face or you have your own data set, you will need to structure it in exactly the same way that we have done here. And as I said in the beginning, Phi 3 mini is an amazing model for its size, so it's a very good candidate for fine tuning when it comes to simple tasks. I'll highly recommend to check this out. Let me know if there are any specific topics that you want me to cover. I hope you found this video useful. Thanks for watching and as always, see you in the next one.