EASIEST Way to Fine-Tune LLAMA-3.2 and Run it in Ollama

Last week Meta released Llama 3.2 which is a new family of four different models including multimodal models. And they're pretty impressive both for language and vision tasks for their respective sizes. But you know what's better than that? It's your custom fine tuned llama 3.2. That's exactly what we're going to learn in this video. We will use Unsloth for fine tuning. Then I'll show you how you can run that fine tuned model locally using Ollama. Because what's the point of a fine tuned model if you can't run it locally? But before then, let's have a quick look at the release blog post. This new release has two sets of models. One are lightweight which is 1 in 3 billion model, and the other set is multimodal with 11 and 90 billion. There is no 405B this time Meta is moving away from the standard 7 or 8 billion models. Now they have 11 and 90 billion instead of 8 or 70 billion model. But I think the most interesting one are the one in three billion models because you can run them on device. We will look at the 11 and 90 billion models for vision tasks in another video. Apart from these models, Meta has also released Llama Stack which is their opinionated version of how developer experience should look. It's great to see that these model providers are now building tech stacks for deployment. Let's talk about how you can fine tune one of these smaller models on your own data set and then I'll show you how you can run this locally using Ollama. Fine tune llama 3.2 we will use the official notebook from the unsolved team. I have covered variations of this notebook in my earlier videos for fine tuning other variants of llama. This is going to be a quick recap of those notebooks. First we need a data set to fine tune the model on. For this example we are using the Fine Tom DataSet which has 100,000 examples. So it's a relatively huge data set. It has multi turn conversations. This data set is collected from multiple different sources. So I think it's a very good candidate if you are fine tuning LLM in general. But if you're fine tuning this model for your own specific task, you will just need to provide your own data set and I later on show you how you can structure your data set. First we need to install nslot. They recommend to use the Nightly version which is basically the latest version. Unslot uses a fast language model class for dealing with LLMs. We are going to load the LAMMA 3.23 billion instruct model. We are using this model because it's a relatively smaller model that you can potentially run on device such as on a smartphone. Another thing to highlight is I'm using the Unslot version. You can also use the LLAMA version directly. You'll need to provide your hugging face token ID and accept their terms and conditions. The 11 billion and 90 billion models are not available in all regions and that has to do with its vision capabilities. So you just need to be careful. Unslot currently does not support vision models yet, but hopefully they will add support soon. When you're loading the model you need to define three different parameters. The first one is the max sequence length. In our case we are setting it to 2048. This number is dependent on your training data set. Look at your training examples and see the maximum sequence length available in your data set. And I recommend to set it to that. But setting it to a higher value will also need more GPU vram. You need to be careful of that data types. You can set it to floating point 16 or 8 but if you keep it none, it will automatically select depending on your hardware. We're going to be using the 4 bit quantization to reduce the memory usage or memory footprint. So here we're loading both the model as well as the tokenizer. Next I'm adding Lora adapters. We are not using full fine tuning even though the model is pretty small. We're adding Lora adapters. These are different modules that we are targeting. We train completely separate modules and then merge it with the original model weights. There are a couple of other things to keep in mind. One is the R or rank. This determines how parameters are going to be in your Lora adapter. If you set it to a higher number, this will give you much better fine tuning or the performance is usually going to be better but. But again you're fine tuning a large number of in your Lora adapter. So that will mean that you will need more resources in terms of VRAM to fine tune or train the Lora adopters. So usually 16 or 32 provides you good compromise between the memory footprint and the performance. Another thing is the impact of this LORA when you are merging it back to the original weights of the model. So that is set through the LORA alpha. Now some points on the prompt template. So here's the prompt template that the Llama 3.1 and 3.2 uses. You need to make sure that your data Set that you're providing in order to fine tune the model actually follows this specific prompt template because we're using the instruct version of the models for fine tuning. If you are fine tuning the base model, you can provide your own template. But if you are working with instructor chat version, then you have to follow the template used by the model itself. The prompt template expects role and content. But here you can see that the data set we are using actually it uses another format which is from human and then I think there is from GPT. Right. So it uses a different prompt template. So we'll need to adjust this prompt template and for that you can use the getchat template class or function from Unslot. Basically we provide the token use the prompt template from llama 3.1 which is similar to 3.2 and that will take all the data sets and convert it to our specific prompt template. So here we are loading the data set. Now we need to go from this which is from system and then you provide the value or from human or from GPT to the role based approach. Everything should be converted to role system, role user and role assistant. We do that through the standardize shared GPT function that we just created. Now if you look at some example conversations here, you can see that we went to the content. So here's the content. Then here's the role. Role is user and that's the question asked by the user. Then we have a role of assistant and this is the response generated by the assistant. When you are formatting your own data set, you will have to follow the specific prompt template in order to fine tune a llama 3.1 instruct version. Another thing is that the llama 3.1 instruct defaults chat template adds this specific sentence in the system instruction. So it's actually telling the model that this cutoff training date was in December 2023 and it adds today's date to be July 26th. So if you see something like this in responses from the model, don't be alarmed because that's just part of the system instruction. And later on they actually mask this. Now, in order to train the model, we are using the TRL library from Hugging Face and we're going to be using the supervised fine tuning trainer because we are doing supervised fine tuning in this case. So we provide the model name, the tokenizer. These are coming from the unsloth. Then we provide our data set. We also tell it which column to use. That's basically our prompt template that we have already formatted. We Added a text column to the data and the maximum sequence length in the training dataset. Now, here are some other specific parameters. A couple of things which I want to highlight is if you set the number of epochs. So for example, if it sets to one, it will go through the whole data set at once, only during training. But 100,000 examples are a pretty huge data set, so that's going to take a while. That's why we set the max steps to 60. You can either set the max steps or you can set the number of epochs. Now, what's the relationship between the two? That is determined by our batch size. In order to get the total number of steps in an epoch, you can divide the size of the data set by the batch size. For example, if you have 100 examples, if you divide it by two, you will get a maximum of 50 steps in the epoch. We're just running it for 60 substeps steps, which is a fraction of the total number of steps possible for 100,000 examples. The reason we do it is because we don't want to run it for a long time. I just want to show you an example, and that's why you probably are not going to see a greatly trained model. In order to get really good training output, you definitely want to run it for a lot longer. The learning rate determines the speed of convergence. If you set it to a high number, the training speed is going to be faster, but the training might not converge. You usually want to find a sweet spot where the learning rate is small enough that it converges, but that will also take much longer to train. Okay, one more thing that you want to train the model on the output, not on the inputs. So that's why you want to calculate the loss of the model on the output from the assistant, not based on the inputs from the user. So the model should see the user input, generate a response, and then compare the output with the original or gold standard output or ground truth. And that's where you compute the loss. So this section takes care of that. It forces the model to only use the output for computation of the training loss or the test loss, depending on if you have a test data set. Now you can look at how the tokenized version of the data set looks like. You can see that we have clearly added the system role. Here is the well formatted user input, and then we have the well formatted assistant response. And this is the data set that we will use to train our model. Want to get rid of this part, which is the system message Part you can mask that. Here we are masking that. And now you can see that you don't really see the original system message. You only see the output the model is supposed to generate. Okay, Next we call the train function on the trainer that we created. You can see that the loss goes down, then comes up again. The reason is that we are running it for a very small number of steps. Probably we can play around with the learning rate as well. That will control the speed of convergence. These are different parameters that you need to play around with. If you're using bigger batch sizes, you can set the learning rate to a relatively higher value. But bigger batch sizes will also depends on the available GPU VRAM that you have. So there has to be a compromise between these hyper parameters that you are working with. Okay, so after this training you can see that if we run this specific prompt on the train model, then here is the response that we get. So here we see the system message, but we will have to mask that ourselves. In terms of the user input. Here's the user input. Continue the Fibonacci sequence. So we provide the Fibonacci sequence and then the response generated by the model is here. Now you can also stream this if you want. Here's an output of the streamed response which basically does the same thing, but in a streaming fashion. Okay. Once you train the model, you can either push this to GitHub or store it locally. I'm mostly interested in how we store the gguvision of the model because I want to load this in Ollama and run it locally. For that to work you just need to call the Save pre trained GGUF, provide the model name. I'm calling it Model 3 billion. Provide the token and the level of quantization. Since it's a relatively smaller model, I wanted to run it in 16 bit floating point precision. Keep in mind this step will take quite a long time because it has to first download and install LLAMA CPP and then convert this model to GGUF format. So here's the model that I downloaded from Google Colab. If you run the training locally, you are going to see UNSLOT FP16GGUF. I downloaded the model from Google Colab. Now let me show you the rest of the process. Next, let me show you how to run that trained file locally using Ollama. Ollama uses the concept of model file, which is basically a set of configurations that you need to provide for Ollama to use a model locally. There are a number of different Things you can use. There's an instruction called from where you tell it which model to use. You can set different parameters such as temperature, max context window and so on. You can also provide the full prompt template for the model. Here is a quick example. If you want to use Llama 3.2 with different configurations than the default, you're going to say from llama 3.2. Here they're changing the temperature to 1. The context max context window is changed to 4096 and you can also provide a simple system instruction if you go to any model on Ollama. You can see this template if you click on this. This is the model file used by any model available on Ollama. I have downloaded the GGF file that was created after fine tuning the model on Google Colab Notebook. You just want to look at the file that is gguf I downloaded here and then created another file called folder define llama in here I'm saying from and then providing that model name with GG at the end. So this is basically the model file that we're going to be using. We can also include a template that will define the prompt template, but since it's already in the tokenizer, so I don't need to do that. You can also define the system prompt, but in this case we want to mask it, so I'm not going to add that either. Now you need to have Ollama up and running. After that you need to provide some details to create this model. In Ollama, we type the command Ollama create. Then what you want the model to be called. So I'm going to call it. Then you can use this dash F parameter and you need to provide the path of the model file that we created. So it's in the same directory. When you click on it, this will start transferring the data and if everything goes well, it will create a model file for us. It's using the template from llama3 instruct. Seems like everything is successful. Now we can run our model. But before that let me show you if this shows up in the model list. So here we have the Find Llama. This is basically the model that we just created. I have a whole bunch of other models that I have already already downloaded. And now in order to run this model, all we need to do is just type ollama run and just like any other Ollama model, we just need to provide the name. Now it's a 3 billion model, so it's going to be extremely fast. If I say hi you can see that it generates responses pretty great and pretty quickly. All right, so I'm going to ask it to write a program in Python to move files from S3 to a local directory and you can see it's really fast because it's just a 3 billion model that is running completely locally and that's the fine tuned model that we just fine tuned. Okay, so this was a quick video on how to fine tune llama 3.2 using onslot and then run it locally on your own machine using Ollama. I hope this was helpful. I'll put a link to the Google Collab in the video. Description in this video I only focused on the 3 billion. Same approach will apply to the 1 billion model for 11 and 90 billion models. The approach is a little different because it has adopter for the vision component, so the same approach probably is not going to apply. But I'm going to be creating some videos specifically focused on the vision model because I think there are some great applications there specifically for vision based rag, which is a topic I'm personally interested in. If that interests you, make sure to subscribe to the channel. I hope you found this video useful. Thanks for watching and as always see you in the next one.