Best Practices For Fine Tuning Mistral

Yes. Okay, cool. Yeah. Yeah, thank you so much everyone for joining this course and this talk. I'm super excited to be here. My name is Sophia Young. I lead developer relations at Mistral. So, yes, sorry, it's June already. The outline of this talk, this talk is we're gonna talk about an overview of Mistral models. We're going to talk about our fine-tuning API that we just released today, and also we have an open-source fine-tuning codebase that you can play with, and then I will show you some demos. Some brief history about Mistral. I hope all of you know about Mistral, but if you don't, we are a Paris-based team of more than 50 people. We were founded about a year ago, and in September last year, we released our first model, Mistral 7B. And then December, we released Mixtral 8x7B with a lot of good feedback on both models. And we also released our first commercial model, Mixtral Medium, and also the platform where you can use our model on the platform through our API. And then February this year, we released Mixtral Small and Mixtral Large. Mixtral Large is our flagship model with advanced reasoning, multilingual capabilities, function calling, all the good things. And Le Chat is our conversational AI interface. It's completely free. So if you like to talk to our models, please sign up and use Le Chat. And in April this year, we released, at the time it was the best open source model, 8x22B. It was, it was really good. And just last week, we released Coastal, which is a specialized model trained on 80+ languages, programming languages, and you can use with various VS Code plugins to, to generate code and talk with your code. And then here's another view of our model offerings. We have 3 open source models, Apache 2 license, which means you can use it freely for your, your personal use cases or commercial use cases. We have 2 enterprise-grade models, Mistral Small and Mistral Large, Mistral Large. Yeah, so for your most sophisticated needs, Mistral Large is really, really good. It supports multilingual function calling, really good at RAG, and now we support fine-tuning on Mistral Small and Mistral 7B. And again, we have specialized post-drill for coding, and we have an embedding model. So we really care about customization. A lot of people asking about like, how do you fine-tune Mistral? A lot of people want to have a recipe or want to use our API to fine-tune our models. So about 2 weeks ago, we released Mistral Fine-Tune, which is a fine-tuning codebase everyone can use to fine-tune our open-source models. And then just 2 hours ago, we announced fine-tuning API. So you can customize your model, customize your Mistral model using our fine-tuning API directly. And yeah, so the technology we use is LoRA fine-tuning. Since this is probably the end of the course, you probably already know a lot about LoRA. It's very efficient. It's very performant. So we did some analysis. The comparison between Mistral-LoRA fine-tuning and a full fine-tuning on Mistral-7B and Mistral-Small. So they have really similar performance. As you can see here, Mistral-7B, yeah, LoRA fine-tuning on this benchmark is 0.9 and the full fine-tuning is 0.91. So very, very close. Note that this benchmark is an internal benchmark normalized to this Mistral-Small number. So Just some note there. And then before I show you some demos, just some thoughts on prompting and fine-tuning. You probably already learned about this. So before you start fine-tuning, you should always think about maybe you can just do prompting. So with prompting, your model can work out of the box, doesn't require any data or training to make it work. So it's really easy and it can be easily updated for new workflows or prototyping. With fine-tuning, sometimes it works a lot better than prompting. It can work better than a larger model, faster and cheaper, because it doesn't require a very long prompt. So for specific use cases with a large model, if you have a sophisticated prompt, you can make it work. But with small fine-tuned models, you can just fine-tune it on specific behavior that doesn't require a long prompt. So, yeah, again, it's better alignment with the task of interest because it's specifically trained on different tasks, and you can teach it new facts and information. Okay, so now I want to show you some demos, basically how to use our fine-tuning API and also how to use— excuse me— our Mr. Fine-Tune, the open-source codebase. Specifically, I want to show you 4 demos. Demo of how can you use a fine-tuned model that may be fine-tuned by yourself or by someone else. Demo of some developer examples, like real-world use cases, and also the API walkthrough and the Mistral fine-tune walkthrough. So, okay. Let me move this bar. Okay, so in this notebook, first of all, we need to install Mistral API. This is the latest version, 0.4.0, that was released today. So if you want to use our fine-tuning features, make sure you have the latest version. And in this notebook, I'm actually using 3 fine-tuned models shared by my coworker. So if you are in an organization, you want to use models created by your coworker, or you want to share the fine-tuned model you yourself created, you can use this, the model from the same organization with different sets of options. So it's really easy for me to use a model that my coworker has fine-tuned today. So for the first example, it was trained on title-abstract pairs from Arxiv. So basically, if you input a title of a research paper, this model will generate an abstract for you. So if I input "fine-tuning is all you need," it will give us some seemingly okay abstract for this paper title. And then just another fun example, the croissant convolution. We made it up, obviously. So the abstract is like, we propose novel convolution layer, the croissant convolution. So it's just a fun example for people to see how the fine-tuning could work. For to play with. And then I have another example, Mistral is all you need. And again, it will output the abstract for a research paper because we trained on the title and abstract from, from the research papers. And another example, here's another model. Note that the model name always have this structure. It's fine-tuned. It's fine-tuned on OpenMistral 7B, our 7B model. So it will always have this name here, and it will have some random strings for the fine-tuned version. So for the medical chatbot, we actually trained on this Hugging Face dataset of AI medical chatbot data. And then here, as you can see, we ask it some questions, and it will answer like it was a chatbot. And another example. So those two examples are from actual data. So here the data we get from archive, and here's the data we got from Hugging Face. But what if you don't have the data? You can actually generate data from a larger model. You sometimes will want to mimic the behavior of a larger model like Mr. Large because we know Mr. Large behaves really well and we want to do some knowledge distillation or knowledge transfer from a larger model. So in this case, if you want to see how exactly we generated the data, you can go to our docs. And in this example, we have this prompt. You are a news article stylist following The Economist style guide. So basically, we want to rewrite the news. As the Economist news article. And then we use Mistral Large to generate the revised news or, like, different revisions of how the news can be revised. And we give it some guidelines, basically. So in this example, we use Mistral Large to generate our data, and then we give it a news article, right? And then we use the fine-tuned data it can generate a new piece of news article that may sound more like Economist news. And then if you give it, give it some prompting, it can also generate a set of revisions it proposed for you to, to change, how you can change the style of the news to make it sound, sound better or more like Economist. Yeah, so that's That's some quick demonstrations of how the fine-tuned model can look like. In our docs, we have some developer examples of real-world use cases that has been using our fine-tuning API. For example, we have, um, Flosflo. They, they are a startup using our fine-tuned model for RAG, for internet retrieval, and You can see the details here. We have examples of RAG for medical domain, financial conversation assistant, legal copilot. So they are all using fine-tuned Mistral models with our fine-tuning API. And in, in these examples, you can see the specific data. You can see the evaluation training steps. And how the benchmark results look like. Yeah, so the fine-tuned was just much better than the not fine-tuned, which is not surprising. Yeah, and then different results. So yeah, if you're interested, you can check out our developer examples for some real-world use cases of fine-tuned model. So yeah, so in this next section, next demo, there are so many demos I wanna show you today because it's very exciting. In this next section, I wanna show you an end-to-end example of how you can use Mistral fine-tuning API on your own and for your own dataset. So of course, we need to install Mistral AI. Make sure it's 0.4.0 or some numbers larger than that, after probably after today. And we also need to install pandas. The first step is, of course, we need to prepare our dataset. Yeah, so whenever you want to do some fine-tuning, the first step is always the data. In this simple example, we read data from a Parquet file that's hosted on Honeyface. It's the UltraChat data. We only read— we are only reading one Parquet file because this data is quite large and we don't want to spend too much money on it. So, so you can feel free to change the data to your own use cases. We split the data into training and evaluation here, and then we save the data locally. Into the JSON-L format. This format is actually needed for using our API. And note that here are the sizes for the training data and evaluation data. So there is a size limit here for our API. For training data, it's limited at 512 megabytes. So you can have multiple files feeding to your training, your fine-tuning pipeline, but each file needs to be under 512 megabytes. For your evaluation dataset, it needs to be less than 1 megabyte. Yeah, so a lot of times the data on Hugging Face is not greatly formatted for the training purposes. So we have a script to help you reformat your data. This script is not— maybe not robust in all of the use cases, but in our use cases it works. So if your use case is more sophisticated, you might want to change the script to your own use case accordingly. So in this example, we basically just skipped several examples because they're not right formatted. So if we take a look at one of the examples, one of the issue here is the assistant message is empty. So we can't really train based on this kind of data. So we need to make sure the data is right formatted. Otherwise, it's gonna be difficult.

I have a quick question about this. Is there like a format validator that y'all have? I see that you have a thing to convert it to format, but curious about validation.

That's a great question. We do have a validator script as well. That was in my next notebook.

Oh, sorry about that.

No, no, no, that's a great question. So we do have a validation data script from the Mr. Fine-Tune repo. Also, whenever you upload the model to— yeah, whenever you upload load the model to our server, it will validate the data for you. So if your data is not right, it will tell you why is it not right and ask you to change it.

So excellent.

Yeah, thanks for the question. Yeah, so yeah, so the next step is to upload the dataset with the files.create function, and then you can just define the file name and then the actual file here, and we're uploading the file. And then we can see the ID of the file, the purpose of the file. The purpose right now is just fine-tune. Maybe there will be some other purposes later, but right now it's just fine-tune. And then to create a fine-tuning job, you will need the IDs of the files we just uploaded. And of course, you need to define the model you want to fine-tune. Right now, we only support Mistral 7B and Mistral Small, but in the future, we might add more models here. And then you can try different hyperparameters. And then in this example, I only ran 10 steps just to have it faster. But if you want to increase your steps, actually, you probably should increase your steps if you're doing something serious. Yeah, so make sure you can change those different configurations here. And then we can find the job ID, and then we can use— and then we can do different things like listing the jobs that we have. I have quite a few jobs that I ran. And we can retrieve a job based on the job ID. To see if it's successful or not. And then you can see we have some metrics here, training loss, validation loss, token accuracy. Because we only ran 10 steps, you only see this once. But if you run it 100 steps, you will see it 10 times. So every 10 step or every 10% of the step, you will see the metrics. So you can see if you are making progress or not making progress. Finally, with the fine-tuned model, when the model is— the job is successful, you will see the fine-tuned model get populated from your retrieved jobs, and then you can call this model and then ask any questions you want. So that's how you can use a fine-tuned model. It's exactly the same syntax as what if you if you are using our normal non-fine-tuned models. Okay, and finally, if you want to use Weights Biases, you can add your Weights Biases credentials here. Yeah, you will need your API key and everything. Just to show you how that might look like, basically you can check your losses, your perplexity score, your learning rate, and everything. So, yep, so that's how you can run the fine-tuning through our API. I hope it's clear. Any questions? Okay, cool. So this last demo I want to show you is what if you want to just use our open-source codebase to fine-tune Mistral 7B or other Mistral models. I'm actually running this in Colab Pro Plus account, so it's possible to fine-tune a model in Colab. So in this example, I just git clone this repo, the Mistral fine-tuning repo, because it's not a package, it's a repository. And in this repo, we need to install all the required packages. And of course, we need to download our model because we're doing everything basically locally. And we're downloading this model from the Mistral website. And we're downloading 7BB-3 model here. And then same thing as we have seen before, we're preparing the dataset. We're using exactly the same data as before, reading the Parquet file. Splitting into training and evaluation, save the data locally. And then same as before, we reformat our data, everything. The evaluation data looks pretty good. And afterwards, you can verify if your data is correctly formatted or not with our evaluation data script. So, yep, so it will— this section will take some time because it's evaluating Yeah, each record. And then for— and then you can start training. The important thing here is actually this config file, right? This basically is telling the model, telling the LoRA how you want to fine-tune your model and different— where is the path of everything. So we have the data file, you want to define the path of your instruct data, your evaluation data. This is the training data, your evaluation data. We want to define the path of your model. We want to— you might need to define or just leave it as default, those hyperparameters. We recommend using a sequence length of 32K, but because I'm using a Colab, it's the memory is limited. I ran into out-of-memory issue a lot, so I decreased this number to 8,000. But if you have a better GPU, you should use 82,000. And then you can define all the other font stuff. Right? Okay. Yeah. And then with this one line of code, we can start training and fine-tune our model. And then in the result, as a result, you can see the checkpoint of your LoRA result here. This is where we can use to, in our inference as our fine-tuning model. And to run our inference, we have a package called Mistral Inference to help you run all of our open-source models and also all the fine-tuning models. So basically, in this, in Mistral Inference, you need to define the tokenizer, which is in the, which you are downloading from the Mistral fine-tune file. We use the v3 tokenizer because it's a v3 model. And then we need to define the model that's reading from the models folder we downloaded. And then we need to load LoRA from the checkpoint that we have, we just saved from the fine-tuning process. And then we can run chat completions and get the result. So that's basically how you can run Mistral fine-tuning with Mistral fine-tune, the codebase. And finally, I want to share some exciting news. That since we just released our fine-tuning API, we are hosting a fine-tuning hackathon starting today to June 30th. Yeah, so please feel free to check out our hackathon, and you can submit from this Google Form. And yeah, very exciting. Looking forward to see what people are building. Thank you so much.