EASIEST Way to Train Custom Reasoning (CoT) Model - From Data Prep to Inference

Let's look at reasoning models and how we can fine tune them on our own data sets. The goal will be to enable chain of thought reasoning through fine tuning. First I'll show you how to generate a data set that enables fine tuning with chain of thought. Then we're going to use that data set to fine tune a small model so that it will be able to do chain of thought reasoning without prompting. The first step is going to be data set generation. Then I'll show you how to format the data. For training the model. We will train or fine tune the model and then I'll show you how you can use that model for inference. We'll be using two different tools. One is Camel AI for the dataset generation and then Unslot for fine tuning. Before looking at the fine tuning code, let's look at the tools. So the first one is Camel AI. It's a multi agent framework that enables you to generate data sets for supervised fine tuning. It can do a lot more than that in terms of functionality. It's very similar to Crewai or Landgraph. You can generate multi agent workflows within Camel AI. And the second tool is going to be UnSlop. It's an amazing package for fine tuning LLMs, especially if you are concerned about the memory requirements and compute needs. I put a couple of other videos that I have previously created on Onslaught. It's definitely worth checking out. This video is based on one of the blog posts from the Camel AI team and here's the workflow. So we're going to get a data set from the user which which will only contain question answers. Then we'll use one of the generators from Camel AI which will convert this question answer data set into a chain of thought data set. So it will implement these intermediate steps using a model from OpenAI. Or you can even use open source models. You can potentially upload that created data set in hugging face for a later usage or to share it with the community. We'll use that data set to fine tune our model using Unslot. And you can also upload that fine tune model back to hugging face so that you enable other users to interact with that model or download it. And I'll show you the difference of responses between this supervised fine tune model on the chain of thought data set and the unfinetuned model on some of the questions that will require chain of thought reasoning. Now we're going to test the model on or train the model on a toy data set. But the steps that I'm about to show you are going to remain the same whether you are experimenting with the toy data set or a much larger data set. Okay, so let's get started. First, we need to install all the required packages. So so we'll need to install Camel AI. We also need to install the latest version of Unslot. And for some reason when I was using the original notebook, I was running into issues with the rogue library. So here I am installing that separately. Now, the main usage of Camel AI in our case is the chain of Thought data generator. If you have a data set that all that already has a chain of thought embedded, you can skip to the fine tuning section. In this section I want to show you how you can use this chain of thought data generator to create your own chain of thought data set from Simple Question answer dataset. For the dataset generation, we will be using OpenAI model. So that's why I'm setting the API key here. But you can also use any of the open weight models. Although my recommendation would be to use a relatively powerful model because you want your chain of thought data to be good so that your model can actually learn from it. Okay, so next we're going to look at the setup of the agent that will use this chain of thought data generator to convert our simple question answer data set into a chain of thought data set. The system prompt that goes to the agent is very simple. You are a genius at slow thinking. Data and code. We'll import model factory model platform type and model Type and then ChatGPT configuration. So first we need to create a model in Camel AI. It's abstracted using the model factory class. You need to provide the model platform Type which is OpenAI and then the model we're going to be using GPT4. All right, you can use the bigger models if you want. Now, there are other options as well that you can choose from. So for example, here's the list of all the models that are supported. So model platform is OpenAI for the OpenAI models, then you can also use Mistral AI, Anthropic Gemini, even the Quinn models. So it's a pretty large model set that you can choose from. There's even option for Ollama VLLM and and Light LLM in case if you want to use open weight models. Next we need to set up our agent. The agent gets the system message, the model that we just created, and then the message window size which is set to 10. Next we need a data set and this is where you are going to provide your own data set. But here we're going to use an example data set that only has two questions. And the first question is related to a mathematical expression. You need to have your question and the corresponding answer. So for example, here's a question, how many A's are in banana? And the answer is three. So you need to provide a data set like this if you want a specialized data set that you want to use as a base for your chain of thought data generation. And we're going to save send this data which only has question and answers into our chain of thought data generator. And this will generate a data that will contain chain of thought reasoning. So we create that object, we pass on our chart agent that we created plus the data set. Now keep in mind in this case there are only two examples. But if you want to find unimodal, you definitely want to have a much bigger data set. Next we're going to use this data set generator which will take every question, the corresponding answer then will generate a chain of thought reasoning. It could generate multiple of chain of thought reasoning and figure out which one gives the correct answer. And once it verifies that, that is going to be stored as the intermediate step. So here this code basically simply tested. So it takes one question and answer at a time, runs through that using the chain of thought data generator that we initiated here generates an answer. Then we use a verifier which basically looks at the question plus the original answer and see whether the answers from the generated data are is correct or not. So if it's correct, then it prints the corresponding chain of thought reasoning that it came up with. So for the original mathematical question, here is step number one, then step number two, then step number three. For the second question, which is probably a lot easier to understand, we want to count the number of A's in the word banana. It's very similar to the famous Strawberry question. So here is how it came up with the chain of thought. So it says, sure, let's break it down. Break down the problem of counting the letter A in the word banana step by step. So step number one is to analyze the problem requirements. Then step two is list the steps to solve the problem. Step three is execute the solution process. So basically looks at the solution that was proposed here. And then step number four is to provide the final answer. Those were just two example questions that we used to see how the data set is being generated. Now we will fine tune our model on this data set. But essentially what you need to do is you need to look at that question Answer pairs. And then we are going to generate a JSON file that will be used for creating our supervised fine tuning data set. And the supervised fine tuning data set is going to contain the the question plus the corresponding answer. Now, since we're using Unslot, so we want to transform this into the alpaca format. I have covered this in a lot of my previous videos, but essentially this is what it's going to look like. So you have the instructions, which is your original question, then input. In this case we don't really need an input, and then the corresponding output, which is this chain of thought reasoning that we just generated. Now, the Camel team also shared this smaller data set which has 12 rows instead of two. And if you look at this, most of the questions are basically counting letters in words, right? So this is going to be the data set that is going to be used for fine tuning our chain of thought model. And in each case you can see the internal chain of thought stream that was generated using the GPT4O model. Now, for your case, you will definitely need a lot more examples than just 12 examples. But even with these 12 examples, we'll see that it can change the way the model behaves. Okay, so here we create that supervised fine tuning data set. If you want to upload this data set to Hugging Fees Repo, you need to go and create a new token and make sure that you give the token right access. If you don't do that, you're going to run into issues. And when you upload the data set, this is how it's going to look like. Okay, so next we're going to talk about fine tuning the model. Again, we'll be using Unsloth because it has the minimal footprint and it's extremely efficient when it comes to using resources. Now you have a number of different models that you can fine tune. For this example, we're going to be fine tuning a Qin 2.2 1.5 billion model. So it's a relatively small model and you can even see that fine tuning can help a very small model with a small amount of fine tuning data set. Before fine tuning, let's look at even a bigger Qin model which is a 72bn instruction. So I asked how many Rs in strawberry? Even with all the training, it's still not able to figure this out. And it says there are two Rs in Strawberry. So we'll see how it performs the smaller model after fine tuning. Now, when you run this, this will download the model. Next we are going to Set up our configurations for LORA adapters. So rather than fine tuning the whole model, we are just adding additional LORA adopters. Most important parameter here is the rank. The rank basically determines the size of the metric that is going to be used for the LORA adopter. In general, you want to experiment with this. 16 or 32 is a good start. But keep in mind as you increase the rank, you are adding more parameters, which will mean you are going to need more compute resources. Now these are the target modules of the model. We set the alpha to 16. This basically controls how much contribution your LORA adopters are going to have on the final trained model. Now we'll keep the rest to default values and after that we need to convert our data set into an SFT compliant data set that we can use for training or fine tuning our QUIN model. We need to have a system instruction basically which is below is an instruction that describes the task paired with an input that provides further context and then write a response that appropriately completes the request. So in our case, we're going to provide the instructions, we're going to skip input and it's going to just generate responses or actually we'll provide the responses so that it can learn from that. Right. So here's a function that will receive a data set or an input example, converts it into the proper format that it needs, and then we're going to use that for fine tuning our model. So in this case I will be using those 12 examples. This is the data set that is already available on Hugging Face. So here's the data set. Now if you look at the data set, it's really focusing on counting the number, the specific letter in the words. So the model is going to get really good at this. And in addition, there's just one example of a mathematical expression. Okay, so next we need to set up our safety trainer, which is supervised fine tuning trainer. So we provide our model plus the tokenizer. It's the same tokenizer as the Quinn model. Then the data set. Now the way this sets up the data, it will add a column text which will have this whole prompt. So in the data set it's looking for that specific column. We define the max sequence then to be 20, 24 tokens, although I think it's going to be relatively smaller for the dataset that we are providing. And then we have all these other parameters which will control the training. I'm not going to go into the details of each one of them. After setting up our trainer, let's look at how much memory we have available and how much memory it's going to use. So on T4 we have about 15 gigabytes of VRAM and with our current configuration it's going to be using only one and a half gigabytes, which is pretty efficient. Although we are training a relatively small model, but I think it still gives you much better resource usage compared to other fine tuning packages out there. All right, so then we start the training process, right? And we're training it for 60 apex. Probably it's a little bit of an overkill, but you can see that the loss decreases pretty nicely throughout training. Now here's what happened. So the training took about four, four minutes or three and a half minutes. The peak memory that was used was only 2.2 gigabytes, which is about what, like 14, 15%? Right. So extremely efficient in terms of the memory usage. Now we can run inference now. So for example, the first question is how many hours are in Strawberry? And if you look at the response, you can see that it's actually using this chain of thought reasoning when it's generating the response. So very similar to our data set, in the response you see that it says to determine how many times the letter R appears in the word Strawberry, we can follow a systematic approach and then based on all the examples that it has seen so far, it comes up with this is these steps, right? So first step is analyze the problem requirement, then list the steps to solve the problem and here's the proposed solution and then execute the solution process. So following the same step that we provided in the training data, and in this case it's able to identify the positions of the words the letters are right, and you get the final answer, which is there are three R's occurring in the word Strawberry. Here's an example of how would you use this trained model from hugging face if you pushed it to hugging face and I'll show you an example of how to push it. But this is the same model that was trained on the data set that I showed you. And in this case you can simply provide the repo ID where you push the model and start using it right away. So I am testing it on a prompt which is bigger, between 9.11 and 9.9. Now in the training data, we didn't had a specific example like this, but the way we train the model, it understands that it has to apply this chain of thought reasoning. So if you look at this, the response here is to determine which of the two numbers is bigger. We can follow a systematic approach. So in almost every response you're going to see this. Then it comes up with four step process. One is the analysis of the problem requirements. Then list the step by step solution that is going to be applied and then execute that solution and provide the final answer. So it goes through a step by step solution and based on that it says that we conclude that 9.9 is bigger than 9.11. All right. Okay. So this was the training process in case if you want to push the model to hugging face. In that case you will just need to save the model locally along with the tokenizer and then provide the repo ID where you want to store the model along with your hugging face token. This will just push the model back to hugging face and as I showed you here, you can reuse that model within unslot once you have pushed it to hugging face. So this was the whole process of how you can potentially generate chain of thought data sets from your own examples using the chain of thought data generator from Camel AI and then use the supervised fine tuning fine tuning trainer within onslaught to fine tune or train your own model. Now on hugging face you might see some data sets for chain of thought which has these thinking tags. So essentially it's really up to you how you format your data set. Just make sure that you provide this in the alpaca format when you are using unslot. Unsoloth also supports other prompt formats as well, but in my experience I have found the alpaca format to be a lot easier to work with. Anyways, I hope you found this video useful. Thanks for watching and as always see you in the next one.