Fine-Tuning with Axolotl

So plan for today, we're going to talk about axolotl, uh, how to use it broadly, and then, um, we're going to go into, uh, the honeycomb example that we introduced last time, and we'll do just a quick catch-up there, um, for those who didn't see last time. But, uh, the honeycomb example, and Hamel Husain will walk through that. We will have some time to get a conversation, both our questions and your questions with Wing, and then we will have some time for Zach to share about parallelism and Hugging Face Accelerate, a very quick run-through of fine-tuning on Modal, and we'll have a little bit of time at the end of this for Q&A. So with all that said, I'm going to get started. The most frequent question that I get from people when they're when you're first starting to fine-tune is, they're really related to, I'm going to call it model capacity, which is how much are we going to be able to learn. The two parts of that are what model should I fine-tune off of? And then the question which is simultaneously more technical, but I think has an easier answer because the answer is almost always the same, which is should I use LoRA or should I do a full fine-tune? I'm going to give a shorter answer. To the base model, and then I'll, uh, walk you through what it means to fine-tune with LoRA. And, um, but then, uh, I think the answer there, despite it being useful to understand LoRA because you're going to use it a lot, um, you should almost always, in my opinion, be using LoRA rather than full fine-tunes. But, uh, the first part of this is what base model do you use? So there are two dimensions to this. So one is what model size do I use, uh, 7 billion or 13 or 70 billion or some other size, um, parameter model. And then the second is, um, what model family do I use? So do I use Llama 2, Llama 3, Mistral, Zephyr, uh, Gemma, whatever else? Um, on the model size, I think different people will have different experiences. Um, I have almost— I've never fine-tuned a 70 billion parameter model. And it's not that we can't, it's actually with— thanks to Axolotl and Accelerate, it's not so, so difficult. But I've fine-tuned 7 billion and 13 billion parameter models. I think most of the use cases I have, the breadth of what we are asking the model to do is not so, so wide. And so my experience has been that fine-tuning a 7 billion parameter model versus 13, actually the 7 billion parameter model like the output quality of these for the projects I've worked on has been close enough that I never felt the need to deal with the parallelism of required for much larger models. So I typically ended up using just 7 billion parameter models. Those are a little bit faster. It's a little bit easier to get a GPU that those run on. And if you look at the download counts, this is not a perfect proxy for what others are doing, but it's some proxy for what others are doing. And you do see that 7 billion parameter models are the most popular. And these are not instruction-tuned models. So these are models that people are typically fine-tuning off of. And you see that the 7 billion parameter model is the most popular. And then for people who want to know just like, what is fine-tuning? We cover that. I cover that in some depth in the— first lesson, so you can go back to that. Then the second question is, which model family do I use? This is one where, again, thanks to the way that it's been abstracted from Axolotl, it is extremely easy to try different models, especially if they all fit on the same GPU, or even if you have to boot up a new instance, that's also not so, so hard, but it's extremely easy to try different models and just do a vibes check. I tend to just do whatever is fashionable. So a recently released model is Llama 3. And if I were starting with something today, I would just use Llama 3, not because I've thought about it in incredible, incredible depth, but rather because it's the newly released model that's widely known to be reasonably good. If you want to find out what's fashionable, there are many places to find that out. You could go to Hugging Face, and then for models, like, just there's a way to sort by hotness and just see what's hot. The LocalLLaMA subreddit is a community of people who think about these things a lot, and that's a good place to look at and just for running models. Though it has local in the name, they spend a lot of time just thinking about different models and how they behave differently. So LocalLLaMA is another community to look up if you want to, to choose a model. But I think people over-index on this and that if you run a couple of models that are just the most popular models at the time, that is, that should be this good enough and you won't probably improve on that immensely by trying many more models. And I'll talk in a couple slides about why that is. The second problem, LoRA-versible fine-tuning, is a question of when you fine-tune the model, are you going to, so you've, let me start with an image. So if we imagine that we've got one layer that goes from an input to the output, I'm going to, for a second, actually simplify the transformer architecture so that we don't think about a query matrix and keys and values. And imagine this almost as just like a, for the, for the moment, a feedforward network. So you've just got one layer that we're gonna look at, and it's gonna take an input that is really an embedding of the meaning of the text up to that point in the string and it's going to output another vector representation. In most of these models, the inputs and outputs are somewhere on the order of 4,000 dimensions. And so just for that one layer, you'd have 4,000-dimensional input, 4,000-dimensional output. So that matrix would be 4,000 by 4,000. That would be 16 million weights. And the idea behind LoRA is that we can learn something that you can add to that original matrix that is much lower dimensional and that, um, will still change the behavior in a similar way but will have, um, many fewer weights. And as a result, it can be fine-tuned on less GPU with less RAM. And the, I think it's safe to say that the vast, vast majority of fine-tuning that happens is either LoRA, or I'll talk about QLoRA, which is going to work functionally in a similar way. But the vast majority that happens is LoRA. And I think for everyone in this course, you should use LoRA for a while. And maybe someday you'll do a full fine-tune, but you as a practitioner, may never need full fine-tunes. There are some theoretical reasons that full fine-tunes, if you have a lot of data, could be higher performance. But, um, and Zach or Wing or Hamel Husain can contradict me here, but I think for most people, LoRA is, uh, all you need. Um, unless you guys want to jump in and correct me, let me, uh, I'm going to say a bit about just how LoRA works. So we want to make some changes to a 4,000 by 4,000 matrix, which is the original weights. And we do that by having two matrices that we're going to multiply together. Those of you who remember your linear algebra will know that if you have a 4,000 by 16 matrix times a 16 by 4,000 matrix, matrix that is 4,000 by 4,000. So if we multiply these two pieces together, that is going to create a new matrix that we can add to the original weights. So it can change the original weights quite a bit, but the number of parameters that are required here— so each of these is— this one is 4,000 by 16, and this one is 16 by 4,000. So if we said how many parameters is that, that's each of these, um, 2 matrices on the right is 16 by 4,000 as the number of parameters. You have 2 of those. So now we have 128,000 weights that we are going to need to fit when we're fine-tuning. That's a lot less than 16 million. And as a result, it just requires a lot less RAM and GPU VRAM. Is frequently a binding constraint as we train our models. And as a result, it's nice to be able to reduce that RAM usage by using LoRA. And you'll see that, yeah, you'll see this just a configuration flag. So it's quite easy to do this in, it's quite easy to do this in Axolotl. The other piece, which is, I think, conceptually also potentially somewhat complex to understand well, but extremely easy to use, is going from LoRA to QLoRA. So here we had each of these matrices, and those are just numbers, or each element in those is numbers. Numbers are stored in computers with a number of bits, and if you store it with many, many bits, then you get very fine gradations of what that number can be. So you can go 2 to 2.00001 and 2.00002 and so on. So we tend to think of those almost as being continuous. QLoRA is dividing the possible values for numbers into a smaller set of values. So for instance, if you start with something that is stored in 16 bits, you can think of that as almost continuous. If it goes— if the lowest value that you want to be able to store is -2 and the highest is, just to pick a number, 2.4, you've got lots of numbers in between there. QLoRA will divide that space so that it can be stored in 4 bits. The number of possible values there is 2 to the 4, so it's 16 values. The exact way that we choose the 16 values is a technical topic that I think isn't worth our going into in this moment. There's some details about how you do backpropagation there that we don't really need to know in practice, but by storing every number in 4 bits, you cut down on the memory usage by quite a bit. And so a lot of people do, a lot of people do this. You'll see again that this is not so complex to do. And in practice, it saves some RAM and it has some small impact on results. But I, I think my intuition would have been that it has a bigger impact on results than I've actually observed it having. And I think most people would agree with that. And so a lot of people run QLoRA models or train with QLoRA either as their default, um, first step, or at the very least it's something that they do frequently. And again, we'll show you how to do that, and it's, uh, shockingly easy. Um, so, uh, maybe it's a good time to just pause for a second.

Uh, Wing, do you— Wing, Zach, even— like, do you have any opinions on QLoRA, LoRA, when you use them? Any observations, feelings? Do you agree? Any, yeah, any further thoughts?

I know that, um, sometimes people see a difference between like the actual losses that, or some of the evaluations that you get during, um, fine-tuning with QLoR because what's happening is you've quantized the weights and then you're training on those, but then when you merge those LoRs back into sort of the original model because the quantization, there's like quantization errors or, um, due to quantization that you're not actually getting the exact same model that you trained. So there has been some like debate over that. I don't like, I personally don't like feel like that's a huge issue. Um, otherwise people would not be using it anymore. So, um, that's really the only thing that I have about that. And I think there was also something that I, I personally didn't understand with QLoRA, what the quantization was. I think there were like double quantization, and there's some like nuances with like that as well when you're quantizing the weights. Maybe if Dan understands that better than me.

I, I think I don't. One of the speakers, so at workshop 4, we're gonna have Travis Adair, who is the CTO of Predabase. Uh, but he built Lorax, which is a serving framework, and he talked about, um, some of the quantization errors as you merge the weights back. I think he has thought about this like way more deeply than I have, and so I know that I'm looking forward to Workshop 4 so I can hear, um, his description of what he's done about this issue. But yeah, I don't know much more about it than that. All of this is, like I said, there's so many places in AI and before that ML where it's like tempting to like get really detailed about all sorts of things that seem very mathematical. The payoff to doing that, even though like most of us were good at math from an early age and we're told like, I used to do a lot of math, fiddling with hyperparameters while sounding cool has a much, much lower payoff than spending that time looking at your data and improving your data. And you might think like, my data is what it is, how can I improve it? And so when we get to Hamel Husain's, what Hamel shows about his work with Honeycomb, you'll see like you actually can improve your data and the payoff to improving your data is so, so large. I think Hamel made a comment about many of you might know who Technium is, but I don't know if Hamel, you wanted to jump in here. Yeah, anyway. Improving your data, the payoffs are massive and you should do more of that. One of the things that we're going to switch into from the abstract, like, hey, here are some ideas to how do we implement this. One of the things that I loved about Axolotl when I switched from using— so Axolotl is a wrapper for lower-level, um, Hugging Face libraries. One of the things that I most loved about this switch from Hugging Face lower-level libraries that give you a lot of granular control to using Axolotl is that Axolotl was so easy to use that I never thought about like, oh, what's the error in my code? And I just spent actually less time looking at code and I spent more time just psychologically looking at my data. And so the ease of changing some things around, um, and being able to run things freed up some mental space for me to focus on my data, which we said is a great thing to do. It also, if you just use the examples, and I'll show you some of the examples, there are a lot of just best practices and default values that are built in. It does a lot of smart things as defaults. I'm going to There are a couple of things that I quite like that it does that we don't have time to cover. And so I'm going to make a couple of videos and then just post them either in the Discord or in Maven or on the Maven portal or both, quite possibly both, showing things like sample packing, which is a quite clever thing that it does that has a— speeds up your training process. But it has a lot of things that you could spend a lot of time figure out for yourself. Or you could just use some of these examples, um, in Axolotl and change relatively few things and have a lot of best practices built in by default. Um, so I, I've loved— so, Wing, thank you. Um, I've loved using Axolotl.

Um, and one thing I want to— maybe it's worth, uh, lingering on for a second is, um, Wing, like, I'll let Wing tell the story. Has there any, have you been surprised by like the level of like, you know, what kind of people are able to fine-tune models, like really competitive ones without like knowing any like deep mathematics or things like that?

Yeah, I mean, I think if you like just sort of, um, like if you think about actually the most popular model, like I think generally like, you know, with Technium's Hermes models and those sorts of ones, like they're generally very popular. And if you actually talk to Ryan, like he doesn't— he's also of the, you know, he's very much like me where he doesn't quite get deep into like Transformers and the math and all of that and just wants to train models and build, you know, focus on good data. So like, really, all of his models are really good. There are people like, I think, like, let's say, I think Miguel Teixeira, is it, with the— I forget which models he has that he releases. I mean, I think his background is more deep learning, but he also uses Axolotl, and there's a lot of like, they don't really need to like go deep into the transformers, right? And so yeah, like Dan was saying, they just are able to spend more time focusing on just procuring good data and doing data synthesis rather than thinking about like all of the everything else that goes on underneath the hood.

Great.

Okay, let's get one level more tactical or concrete. So using Axolotl, we are— some people here have used it a bunch. We're going to make the assumption that most of you have either used it very, very little, or I think even more when we did a survey at some point of some students, most of you have not used it at all. So this is going to be really a how do you just actively get started. It's— I think you'll be surprised that it is not so, so difficult to run your first jobs, and I highly recommend doing that. You'll just feel different about yourself as someone in this space once you've run a couple jobs and you feel like a practitioner now. So highly recommend using it. Um, the way to get started is if you go to the Axolotl— actually, I would just start with just Googling GitHub Axolotl. If you go to the Axolotl repo, there is a separate documentation page, but just the README is fantastic and has most of what you'll need. I'm going to point out a couple of things that you should look for while you are at, in that README. So the very first is examples. I mentioned earlier that there are a lot of examples. Axolotl takes YAML config files, and the config files are reasonably long. Uh, maybe Wing could do it, but I don't think there is anyone else who could open up them or have like a blinking cursor and then just type one out beginning to end and get it right. So you and almost everyone else will go to one of these examples, copy it, uh, the first time you should just run it, and I'll show you how to do that. But then you're likely to change one or two parameters. Probably the first one that you might change is the dataset that you use, but you might change one or two other parameters, rerun it, and it will always be an experience of taking something that works and then changing it around a little bit rather than starting from scratch. So you're going to use, um, these examples, uh, to show you one of them. So here's one. This is, uh, to fine-tune a Mistral 7B model with QLoRA. So the first, the very top is showing you what is the model that I'm fine-tuning off of. So this is QLoRA. So here we are loading in 4-bit. We have a dataset. I'll show you that dataset in a moment. We're going to store the dataset after the prep phase in some location. We're going to have some validation data. Um, most of these you won't change that frequently. Sample packing, I'll make a separate video about. Uh, this LoRA R is related to the size of those, um, LoRA matrices where that's that matrix that I was showing earlier. LoRA alpha is a, a scaling parameter. I, I wouldn't worry about some of these bottom ones. I think the ones that you probably want to focus on up front would be actually It's not the easiest one to change, so you could change something else just to get an experience of changing it. But when you really start working on your own use cases, the first one you'll change is the dataset. And the format of the dataset is— so there are a lot of different formats. I think one of the things that's really nice about Axolotl is that out there in the wild, Data is stored in a variety of formats, and if you tell Axolotl what formats it's stored in, you can use most of, if not all of the common formats. So this is a format called Alpaca, but each row or each sample has an instruction to the model, optionally some input you'll see in these, most of those are empty. It has the output, which is what we want the model to learn to reproduce, And then it has some text which will go above these. So the text would be below is an instruction that describes a task, blah, blah, blah. And then you'll have a question like, what is the world's most famous— who is the world's most famous painter? And then here's the training output, which is what we're going to train on and try and have the model learn to replicate the behavior of.

So just to kind of stop there for a second and talk about the config files. So like, when I start a project, I, you know, I look at the examples too. I message Wing sometimes. Now, not everybody can message Wing. Please don't message Wing with like, not, please don't, don't DDOS him with questions like that. There is a Slack channel, an Axolotl, sorry, a Discord channel. I think Wing looks like he's getting the link and putting it in the Discord right now. Um, and that's a good place to like kind of trade configs. But yeah, starting with a known good config is a good idea. It's like, hey, like I'm training this model, uh, that just came out, does anyone have a config? And usually either by searching that Discord or looking at the examples or something else, you can find a config. And there's a lot of times in Hugging Face repos nowadays you can find Axolotl configs as well. Wing, do you have any other tips on where to find configs or where, how people should kind of go about it?

Yeah, depending on like some model creators, I know like personally I try and include sort of the model configs when I'm releasing models either somewhere in the repo or in the README. I think Axolotl, by default also stores in your README, it'll store it, the Axolotl config. So sometimes, like, if you go through Hugging Face and there is a link where you can find, like, models that are tagged that were trained by Axolotl, depending on whether or not they modify their README, you can sort of, like, get configs from there as well. But other than that, I think a lot of times it's, you have, you'll see some examples in the Discord people have and I'm happy to also help just like, you know, with various things depending on like what, but it's generally pretty self-explanatory most of the time, I think. So yeah, usually you're wanting, you're taking like little bits from like one config and maybe combining with another piece, whether it's like FSDP or DeepSpeed or, you know, the LoRA versus QLoRA versus like, there's a, most of the, most of the various configurations are pretty composable with each other. And if they're not, I believe we do enough validation that, it will tell you that it's not composable.

Sounds good.

Yep. Okay. And then a lot of those, a lot of, there are a lot of other parameters. I won't go through these in, I won't go through most of these. Most of them you won't change, but I will say a couple of things. One is, Many of us like using Weights and Biases. It's a very nice Weights and Biases integration in Axolotl. You'll even see a config from Hamel Husain later on that shows you how to fill this in. Micro batch size is just the basically batch size per GPU. Yeah, and then a lot of this stuff you won't change in the near future. And so like I said, I highly recommend starting with any of the example configs and then changing it just small pieces. Don't get overwhelmed by all the things that you aren't changing. Then once you have your config, the next step is to run it. Um, like I said, I think the, uh, this GitHub README is so, so useful. So after you've got your example, uh, click on the Quickstart section, and that will bring you to a set of, uh, depending how we count, either 3 or 4 commands. So the reason this, while it looks like 4, could be 3 is that there are 3 steps. So one is preprocessing your data, the second is this training step, and then after that you're going to want to just test out your the, the model that you've trained. So there is a CLI tool to do that. That's this third step, and Hamel Husain will actually show another way to do this. The thing that I like to do is there's also, if you run this bottom version instead of the third, that launches a very lightweight Gradio app so that you can just on in the web type something into a form and that gets sent to the model and inference happens. And then the output is shown. So I quite like using this bottom step. You will—

I think it's worth mentioning, you only want to do this to kind of like spot check your model. This is not for like production. You don't want to do inference necessarily in production with this.

Yep.

And we'll cover inference in production in the deployment workshop. Sorry, I lost my train of thought. Ah, you, so you will not remember these commands. The thing that I hope you remember is that everything you want is at the, is in the GitHub repo and this one is in the quick start, but it's just the series of commands. So what does it look like if you run that? I'm gonna show you. Some of the text here is gonna be relatively small, So we'll come back and I'll show you a screenshot that you can see some stuff in more detail. But this is just a very quick view of what happens when you train the model. So actually, I'm going to make sure that you can see it in reasonably high depth. So here I am typing out that first preprocess command. I use the debug flag. Hamel will talk about the debug flag, whether you use it or not. When he gets to his section, but I kind of like using it. And when you do that, there's some output here. In a moment, I'm going to go into that in more depth. And then after that, I run the next command that was shown on that last screen. This is just doing training. And that kicks off training. And then training, depending on the amount of data you have, can take Yeah, minutes, hours, I suppose sometimes days though. The projects I do, actually I do have one project where it can take days, but it's typically, you know, an hour or so and sometimes much less. So let me go to the next slide. In there, there was a section that it printed out. From the preprocessing step with the debug flag that it would be easy to overlook, but I think is really critical for you to— in your understanding of what is happening here. So though we started with data that had multiple fields, your model is going to train on a string, or I'll show you in a moment it's actually a string and one other piece, but it's going to train on a string. And so This is showing you the template for what does that string look like that we, that we create in the preprocessing step and then that we later use for modeling. So we have, say, there's an instruction and input and output. And actually those are for each sample just filling in. Here's the instruction, here's the output, here's the text. When you use this for inference, you're going to want to provide everything up through this response part, but then not the output 'cause you wouldn't know the output when you use this for inference. But this template is showing you what the string looks like. And then we're going to use that autocomplete type logic so that we provide everything before the output and our model will provide the output. This actually, this looks like it's just a string. There is one other piece that I think is important for your understanding of fine-tuning that is shown here. So it's actually a string and a mask. So I'm going to go back here for a moment. When you calculate your loss function to, which is part of, for those of you who are familiar with deep learning, which is just part of figuring out how do we change our parameters to change the model's behavior. We don't want to train the model to write the words below is an instruction that describes a task, and we actually don't even— the input here is a proxy for what the users of your app's input will be. So we don't want to train the model to be the user. We want it to instead be good at responding to user inputs. And so these pieces up front are not going to inform the loss. So when we look at the, um, the output, we can look at it on a token-by-token basis. So somewhere in there, there was some input, and there were the words that appropriately completes the request with a period. Each of these are tokens, and before that we have pairs of the word that is token ID 2899. But because we don't want it to feed into the loss. We have the first piece of this tuple here is -100, which is just a way of preventing it from influencing the loss and thus influencing the behavior of our model. If you look at the output that's in green here, and for those we have the token ID, then we also have the purpose of calculating a loss, what token is this, and it's the same. So There is a flag, which I think is called train on inputs, that will let you change this behavior. But broadly speaking, this is just showing that there's a way of being able to see very clearly which— what are the tokens that are influencing— that are the inputs to the model, and what are the tokens that are influencing loss and that are eventually going to be the outputs of the model or that we're training the model to output.

Wing, do you use that debug thing?

And yeah, all the time, because, mostly because I want to be sure that the tokenization is correct, because a lot of times I'm using ChatML. And so like, because it's not a default token, I just want to make sure I didn't mess anything up in sort of setting those special tokens for ChatML. And just to double-check that, you know, the outputs look right.

Just so people know, ChatML is a specific type of prompt template. So if you go back to the previous slide that Dan had, you know, that this I believe is an Alpaca template. This is Alpaca. Yeah. So that's a specific type of template. And yeah, ChatML is different.

And in general, chat templates tend to be a little more— there's a slight complexity or nuance to them that instruction tuning templates, I think, arguably are a little simpler. But Okay, so I didn't mean to cut you off, Wing.

We can keep going.

Yeah, no, I mean, that was really— and then sort of like checking sort of like the end tokens, making sure that sort of the stop tokens are in there correctly. And just because if sometimes if it's not in there, you can get a model that just starts to like ramble on and on and never stops. So it's just a good like spot check for myself and sort of especially multi-turn conversations just to make sure that it's like masking out the responses correctly. And you can sort of see that because it'll go like red, green, red, green, red, green. So yeah, it's just an easy spot check. And the color— having the colors just makes it easy to like just glance at it just to like without having to like— because that is hard. That is actually really hard on the eyes to try and debug. So yeah.

Cool. Let me show this, this last step. So we've done training. There was one more command. I'm gonna show the Gradio version of it. So let me pause this for a moment, then switch over to make sure that we're looking at this in the highest possible resolution. So the last step was to kick off the app. I'm gonna run this accelerate launch. Have the inference command with pass in the right YAML file, the directory with the LoRA, and then this Gradio flag. This kicks off an app. You can click on that link, open something in the browser, and you can type and test, test things in the browser. So that was that last step. Again, you won't remember all of these pieces, but you should remember that they're in the Quickstart and you can refer back to this and Again, super highly recommend before other things get on your to-do list that you run through this so that you have hands-on experience using Axolotl. With that, let me hand it off to Hamel to go through a case study, which is the Honeycomb case study. You want to— Hamel. Um, you want to take over sharing?

Yeah, let me do that right now. Okay, see here, let me start the slideshow. Is that sharing good? Okay, thank you. Okay, so, um, we covered the There's a through example in the, in the workshops, in the fine-tuning workshops, and that's this use case of Honeycomb. And, um, we discussed it in the first workshop. Because we have so many students, I'm going to go over it really quickly again. So the case study is you have, uh, there is a company called Honeycomb that I've worked with, and Honeycomb is an observability platform. It's a telemetry system that allows you to log all kinds of data and it tells you things like, it helps you diagnose like if parts of your application are slow or there's bugs somewhere like that, uh, or something like that. It's kind of like similar to Datadog in some ways. Um, Honeycomb has a domain-specific query language, um, called HQL. Um, and one of the things they want to do is like reduce the burden of people learning HQL. And so what they did is they released an alpha product that allows users to type in natural language queries. So instead of learning the Honeycomb query language, you can just type in your question. And so the way it works is you have two inputs to the LLM. You have the user's query, and then you have the user schema. The user schema is retrieved with like a RAG type approach. We don't have to get into that. So with these two inputs, there's a prompt, and then out comes a Honeycomb query. So that's like the, that's the sort of high-level overview just to remind you. So let's jump right into the case study. For the case study, I'm just going to be walking through some slides. And let me, let me open this GitHub repo. So it's github.com/parlance/labs/fd-course. You don't have to open it right now. Actually, just would follow along with what I'm doing. Is this repo, right? Or it's a— so I'm going to open— actually, let me open the repo. So just to show you, so it's a repo that looks like this. I'm just going to go through the notebooks. They're numbered 1 through 8. And Dan, tell me if you can see the text on my screen or is too small or whatnot.

I've got a big monitor, but it looks really clear to me.

Good, Zach. Okay. Um, okay, so let me just— I'm just gonna go through some steps. These steps are not necessarily linear, but it'll give you a good idea I'm going to be focusing a lot on what we did with Honeycomb to fine-tune a model. And a lot of the steps are going to be around dataset curation and, you know, data filtering and debugging and evaluation. Because, you know, we're not, as Dan mentioned, we're not really focused on the model so much. And so basically, I just want to go through the prompt real quick. So this is the Honeycomb prompt. It's basically the system prompt, Honeycomb AI suggests users queries. This is one of the inputs. This is the schema. There is this long fixed part of the prompt, which is a query specification, which is just like a bit of a programming, like a very terse programming guide to the Honeycomb query language. There's some tips. And there is some few-shot examples of queries, of questions, or, you know, user queries and then Honeycomb queries. So there's a few-shot examples. And then finally, this is a completion model. So when Honeycomb launches, they use the completion API. And so the chat API. And so, you know, they're just completing this based on the user's question, which is templated. So the interesting thing is, so you know, they— you could see that there's a lot of stuff in this prompt. Like, like, all of this stuff is fixed every single time, um, in this particular situation. So you're like, you know, the few-shot examples plus the tips plus the, um— sorry, I didn't go over the tips. The tips are just like additional instructions. So all of this stuff is fixed except for the columns in the question. So that's a lot of boilerplate to be sending to large language model. But then also, it's like, it's hard to specify everything you want in this prompt. Like, no matter how hard you try, you hit a wall. And like, that's where fine-tuning, uh, kind of moved the needle. So Honeycomb launched this product. Uh, here's— there's a link to the blog post. Um, it's kind of neat to read it. Um, and Yeah, it just talks about the same thing. You type in a natural language query and out comes a Honeycomb query and you can read about it. I don't wanna go too deeply into that. So the goal in this case was to encourage more users to write queries. So like the bar isn't like super high in terms of like it has to be perfect. But one thing we had to do is write evals. So like one of the things you should think about is writing evals. After you, after you do kind of like some prompt engineering, you may like prototype with a large language model just off the shelf if you can, just to see if like, just to get an idea of how well it works off the shelf. So with Honeycomb, so what do I mean by evals? So I have this blog post about evals. I won't go through it in too much detail, but there's different levels of evals. Level 1 is unit tests where you write assertions, and then there's Level 2 and Level 3, and I'll be going through like the, like, you know, Level 1 and 2. Level 3 is A/B testing. So basically the idea is you want this virtuous cycle where you have evaluation at the center. And, uh, the Honeycomb example is actually like a really good use case because it's like very narrow and like simplified, and, um, it kind of like allows you to like get what I'm talking about. So basically, like, you don't have to understand this code, but just know that for the level 1 evals— when I'm talking about level 1 evals, I'm talking about assertions and unit tests. That don't involve calls to a large language model. These are like rules that you can think of that you can run almost instantaneously and get feedback about whether your model is doing the right thing. Okay, and so there's some code here, and I'm just showing you this code so you know that it's real in case you want to see an example. But it's essentially what I'm doing is I'm just testing different things about the Honeycomb query for correctness. Okay, I'm like testing this valid JSON. I'm testing if it's— there's invalid columns in the query. Based on the schema, if there's invalid filters, you don't have to know the specifics of this. Just know that there's lots of different level 1 evals. Okay. And you don't necessarily need to write it like this, but just giving you an idea that you need to write these assertions. And also, like, so just let know also that I had to iterate on this quite a bit. Like don't expect that you're going to get all the assertions right the first time. There's an iterative loop where you kind of, you know, throughout this whole process, you have to update these level 1 evals. You'll notice more and more failure modes. And I had to work really hard on this to get something that I was happy with. And then, like, you also want to use these evals. You want to write them in such a way, these assertions, that you can use them different places. So you not only want to use it for tests, you also want to use these evals to filter out bad data for fine-tuning. You want to use it for curation, and you also want to use it in inference so you can do self-healing. Um, and so like, you know, I have like encapsulated this query checker. Again, you don't have to know what this is, just gives you an idea like, hey, I'm using these like assertions in different places. Um, and this is like an because this use case is oversimplified, this kind of way of organizing your code may not work for you. You have to do what works for you in that situation, um, but just know that it's here. Okay, um, and already went over this: assertions are not just for tests, they're also for filtering and curating and inference. And yeah, definitely look at the blog post. Okay, so one thing that you will often have to do when you're fine-tuning is like acquire data. And a lot of times, like, you don't have the data in an applied use case. Um, so what do you do? Like, in the Honeycomb, in real life, um, my counterpart Philip, who I was working with, didn't have lots of data. He launched this to, you know, uh, production. But then, like, you know, not only did not have lots of data, a lot of that data was private, and I can't see that data. And so what we, you know, he gave me about 1,000 examples and I wanted to set aside a fair amount of those examples, like in the eval set. So I, you know, so I could test the model. So I wasn't really left with much. And so like the question is, okay, what do I do from here? Like, so you all, you're, a lot of you, if you're in the wild, and you're trying to build something with large language models and you're trying to fine-tune it, it's good to know about how to generate synthetic data. There's no hard and fast rule, again, about like how many examples you need. I just generate as many examples as I feasibly can, just based on intuition, based on like how much it costs, how much time it takes. I ended up generating 30,000 examples synthetically, but I kind of went overboard. So you don't have to do that. Just use your intuition based on your budget and what you have. So like, you can do this with prompting. So let me give you like a concrete example. Because if I just say, hey, you can use a large language model to synthetically generate data, you're like, well, how? Like, what does that mean? And I think for every use case is different, but let me show you what we did for Honeycomb. So the prompt is basically the same exact prompt that you've seen before, except there's a second part that says, okay, you are given the following 3 inputs: a natural language query, a list of candidate columns, and the query. Your goal is to generate correct variations of the combination of NLQ, candidate columns, and query to build a synthetic dataset. You can build a synthetic dataset by rewording the query and substituting the column name response should be JSON with the following keys, um, so on and so forth. Um, and then basically, yeah, I'm giving it the inputs now and then saying please basically perform data augmentation. So substitute, like rewrite the natural language query, substitute the columns, and substitute the query. And basically I'm able to generate lots and lots of synthetic data this way. Now, you might be wondering, is that good data? Like, is it duplicated? Like, all this stuff? Yes. And you have to clean it up, which I'll talk about in a second. But just know that, like, for example, you want to use those level 1 assertions as your first line of defense. A lot of the stuff that's going to come out of this is going to be junk, maybe, or some amount of it. You want to get rid of it. So the level 1 assertion is already going to help you. It's going to help you throughout this whole thing. Um, okay, so you have a way of getting lots of data. This is how you do it. I'm not going to show you the code of doing that. It's fairly straightforward. It's like, use your favorite large model to do this. Use the most powerful model you feel comfortable with to help you generate the synthetic data. Um, and then, okay, so the next step in this is like preparing the data for Axolotl. Um, we're gonna So like usually what I do is like I go through a run, I run all the way through and I see like kind of what's going wrong and then I come back and improve it. You know, you don't want to just like try to make your data perfect the first time and then like, you know, go through it. You want to like go all the way through, see some predictions, make sure the plumbing works, etc. And then you can come back and curate and filter the data. That's what I recommend because you can get stuck, it's good to know where the problems are and have an idea. So, uh, okay, so you want to prepare your data to look like this. In this case, because I'm using the ShareGPT, uh, Alpaca format, um, and I'll tell you what that means. Basically, if in Axolotl there's this config ShareGPT and Alpaca, um, And let me just open the docs so you can see that. So there's the dataset formats. This is the Axolotl docs. There's different formats. We're going to— I'm using a conversation format, and there's a SharedGPT. And you can see SharedGPT, you have to structure your data like this. You have conversations, and then you have from and value, and you have different roles. Like the from can either be human or GPT. And then the value. You can also have a system prompt, which I do have in this case, um, which I'll show you. But anyways, like, you can see that follows that here. I have this, like, uh, conversation where I have a system prompt, then a human, then GPT. Now, why is that? Uh, well, that's the way that Axolotl expects your data in for this format. But also, it's important because If you remember Dan talking about the train on inputs, you know, not training on inputs. So this is considered an input, the system, the system role in the human question, it's considered inputs. And the output is considered is this at the query. And so what we're doing is we are only penalizing the model, we're like forcing the model to basically learn to get the right query and not trying to have it predict what the question is, if that makes sense. So you organize your data like this to this JSONL, um, and let's take a look at the config. So the thing you want to pay attention to here— Dan already went over the config, but in this case, change the dataset. This is a local dataset. So I have this basically the sample data and I have this like synthetic queries, um, and you can look at what that looks like if you want. It's in that GitHub repo at this path, um, and then also the train on inputs is also false. There's a, there's a key in here, train on inputs, which I'll let you find. I don't want to try to hunt for this right now. It's right here, train on inputs. And then also you want to change— if you're going to run this example, which you can, and I'll show you how, you need to change the following things in your config. Like, you, you won't be able to access my Weights and Biases account, and you won't be able to access my Hugging Face account. Probably want to create your own. And so, like, what Axolotl does is, like, you can log— as Dan mentioned, you can log all the training metrics to Weights and Biases. And then also you can also put it in a Hugging Face model repo and it will upload your model to that repo, which is super handy. You know, it'll do that at the very end. And I'll show you what all this— I'll show you some examples what this looks like. Okay, so prepared the data, you got your config file. Now what do you do? So what I like to do is I don't ever jump straight into training. Ever, because I'm dumb and I make a lot of mistakes in dataset preparation. Always make like do something wrong. And honestly, I think a lot of people do something wrong here. And so what I like to do is look at the data and I look, I like to double check how Axolotl is preparing the data. And the way I do that is I do this axolotl preprocess command. And that will basically flatten the data and assemble it in the right format. You can see all the different commands by using help. So I just show that here just for reference. And so I like to look at the data manually. There's that debug thing that Dan showed, but I like to like look at it manually just so I can like kind of play with it a bit more, manipulate it, kind of inspect things. So basically what happens is when you preprocess the preprocess the data, Axolotl dumps that data by default into this last run prepared directory, and that is a Hugging Face datasets format. And so you can load that Hugging Face dataset format and inspect it. And that's what I'm doing here with this code. Basically, you can see it has sort of flattened that JSONL into a format that looks like this. And that is the Alpaca format. Just like how Dan showed earlier, you have this like instruction, um, and then response. And so, um, yeah, like what I recommend is check multiple examples, make sure it looks right, make sure you didn't put the wrong thing in the wrong place or have like things in there that you didn't intend in your data. Happens all the time. Um, one thing that I'll mention is, yeah, there's these spaces right here. You might be wondering what the hell is that. Um, it's a little bit of a tricky issue. It's kind of some artifact about the way Axolotl assembles, um, you know, tokens. Um, I don't know if Wing wants to say something about this yet, but I found it not to be an issue as long as you're like consistent with inference time. And I'll talk more about that. And I have a blog post about that as well. Okay. There's also verbose debugging, which Dan already covered, you know, and basically, yeah, you could just debug flag. The special tokens are here and that's like worth paying attention to. But there's like the red, green. I'm not going to go through that again. Um, and then yeah, it's always good to know what, like, the spot check, like, what these tokens are and if it's correct. So like, for example, like, what is this token? Like, you might be wondering, see this, you haven't done this before, you're like, what the hell is that token? Is that wrong? Like, okay, that's a newline. Um, but yeah, if you want to go into like why, what's going on with the tokens, there is this blog post here. Um, I'm not going to go through it now. But just tokenization gotchas. As an exercise for y'all, you might want to go through this blog post as homework and take a look and see, you know, if it's something that you find that matters. I was really super paranoid about these like small things like spaces, but I found that it didn't matter. And I actually discussed this a lot with Wing. But Wing, do you want to— do you have any opinions on this? Is he here? Might not be here. No worries. Okay, I'm just gonna go straight on to the next. So, okay, that was dataset preparation. Now we're gonna talk about training. We already seen the config file. The config file is also located at this location. Which I will go through. You can see it's been uploaded to Hugging Face. There is a link in the notebook, so you don't have to memorize what you're seeing on my screen. To run training, you run this accelerate launch axolotl command, and Zach is going to be talking about Accelerate. I don't want to go into that deep rabbit hole right now. Um, I'll just let Zach talk about Accelerate in a bit. If you notice, I have a Weights and Biases config here, and this Weights and Biases entity is just basically like a GitHub org, and the project is basically like the repo. And so when you do that, Axolotl will upload— you can log your training runs to Weights and Biases. Let me show you Weights and Biases real quick. So Weights and Biases looks like this. It's a bunch of runs, and you can, you know, Yeah, you could just log your runs and the results, look at your training loss curves. I'm not going to spend too much time on this, but just know that it's there if you want to look at it. So basically, like with training, what did I do? I tried different parameters. I varied the learning rate. So first of all, I took a— so this is Mistral 7B. So I went into the examples. I asked in the Discord. So on and so forth. Like, what is the best, uh, what's the best config for Mistral? And, um, you know, I started with that. And so I varied the learning rate, I tried different learning rate schedulers, I actually tried like different distributed scheme schemes, like using DeepSpeed, like DeepSpeed 0, 1, 2, 3, just to, just to test stuff. I mean, not that it matters, but, um, uh, actually this is a small model, so it fit on my GPU just fine. Um, but yeah, I mainly just vary the learning rate and the batch size. Um, another thing is like, you know, there's sample packing that you might want to try, um, to save GPU space, um, or to like save the amount of VRAM you need, or like, you know, increase throughput. Um, but Dan will upload a video for that or talk about that in a, in a little bit more detail later on. So when the training is done, it's uploaded. If you put your Hugging Face ID, it's uploaded into Hugging Face, which is here. So this example of this model is here. You don't need to know what is here. I don't want you to kind of— you can look at this later. And I'll go through like some of this code in a bit. So the next thing you want to do after you train your model is to sanity check it. Okay, and like, um, there's a lot of different ways you can sanity check your model. I like to, uh, you can use the way that Dan mentioned earlier by using Axolotl directly. However, um, I like to actually use code to up to like, um, and using Hugging Face Transformers to actually, uh, make this work. Hey Dan, I think like, uh, Wing may be trying to open his camera potentially. I don't know. Uh, um, okay, so sanity check the model. This is the Hugging Face repo where the model is uploaded into. Don't be confused that this says like Parlance Labs and the other config says Hamel Husain. That's because I changed the name of the repo and I didn't want to break the links. But, um, yeah, so this is just code about basically pulling that model from Hugging Face. And then this is the, this is the template. So another reason why I sanity check things this way is I want to make sure that I understand the template and that it works. Because I had my own, like, basically, yeah, and like the way I want to do is I just want two inputs, a natural language query and the columns. There's different ways to do this. You can use Hugging Face has like a templating system that you can use. I'm not going to go into it. But I'd like to, like, make sure I understand the template. Um, and so that's what I have here is I have this template. It's basically the same thing. Um, and this is just code to, like, run it. Um, but basically it's just, like, sanity checking examples. Okay, so nothing too crazy going on here. I just have some natural language queries and some schemas, and I'm checking to make sure that it works. Um, that's what you should do That's the first thing you should do. Okay, great. So we've done all this stuff. We trained the model. We sanity checked that at least like the plumbing works and some results maybe look plausible. So the next thing you want to do is like, so the question is like, is this any good? Yeah, it passes. Like you can see like these level 1 evals, you can track the different metrics of the level 1 evals. You can know like which assertions are failing, how, you know, like what kind of errors are you getting the most? That's all good. Um, but then like beyond the level 1 assertions, after you conquer those, like, are these like good or bad? So when I, when I shared— so I launched this model onto Replicate for inference, and we'll go through inference later, so don't want to like get stuck on that— is like, uh, you know, and allowed— it did some sanity, more sanity checking, and, um, basically like fill up did some sanity checking and said, okay, this model is okay, but it's not great. Um, it's still making some mistakes in some places. And actually, it turns out that the data that we used to expand, um, that data wasn't great either. And this will happen all the time. Um, and you might, you might find this when you're doing, uh, and like basically you have to do some error analysis and figure out like, okay, if a result isn't great, uh, like why is that? And one way to do that is like to look at the data, look at the training data, try to debug. Like this, in this case, I looked at similar queries in the training data and try to see what was happening. And we found that, okay, like actually the training data could be better. Um, you know, like things are passing the level 1 test just fine, but they're not like the greatest queries. They're syntactically correct. And so what do we do now? So like one, one thing you might be wondering is, okay, like, are we stuck? Do we have to stop here? Like the data is meh. Like, and Philip doesn't have time to sit there and label a bunch of data or write better queries because he doesn't have time. So what do you do now? Okay, like what you can do is basically you want to try to encode the knowledge of Philip and his opinions into a model. Like you want to like see like, can you have like Philip as an AI in this situation? So what I did is I started building an LLM as a judge. And basically it's the same exact original prompt, um, but basically in like that you've seen before, but with an instruction that you are going to, uh, be a query validator. Okay, you are an expert query evaluator that has advanced capabilities, judge the query good or not, blah blah. And then there's a bunch of few-shot examples here of Uh, you know, like inputs, NLQ, columns, query, and critiques. And basically what I did is, um, I did a bunch of— so how did I get this? Um, in this case I used a very uncool, low technology technique by using a spreadsheet, and I sent Philip a spreadsheet every day for a few weeks and had him write critiques. And over time, what I did is I aligned the model as much as possible with Philip so that it was agreeing with him in the critiques it was writing. And I kind of kept tweaking the few-shot examples and the instructions until I was— until we were both satisfied that this LLM as a judge was doing a good job. and the thing that was really good about this is like, and so I talk about this a little bit more detail in the blog post when we talk about level 2 human and model eval. Um, I don't want to go in, there's a lot you can say about this, like there's different ways you could do this. I just want to give you an idea so that you have it like the general process in your mind and you know that this is a tool in your toolbox. Um, it's impossible to teach everything I know about it in one, you know, in such a small session. But what I'll say is, uh, yeah, like when you have the result of this, you get a bunch of critiques and, uh, you can use those critiques to actually make the data better. And you can use the, you can use the same LM as a judge to filter and curate the data, like filter out bad queries. Hey, like try to make the data better. Given a critique, can you make the query better? If it still can't make the query better, then you filter it out. Um, so that's kind of like a sort of, you know, what we, what we went through. Um, and so basically from there you can curate your data. So like what I mentioned before, First thing is you can like fix the bad data again using a large language model. It's like you're giving the following inputs in a critique and then it's output the improved query and just output the improved query. That's one way you could try to like increase the quality of the data. But then also you, like I mentioned, you want to filter the data. There's many different ways to filter the data. When we talk about dataset curation, there's a lot of things that you can do. Um, uh, in like filtering, again, you want to use both your level 1 evals that I mentioned, like those assertions. You want to use these level 2 evals to do even more filtering. But then also you commonly have other filters that you will find, uh, that you, you'll see like different things in the dataset. You're like, oh, like things in this part of the dataset are garbage, or like, hey, the model is making a certain kind of mistake. Let me, let me filter that mistake out. And then you have to decide whether or not you have to go acquire data for that mistake. So one example of that, um, that's not necessarily a test but it's a filtering technique, is in this case I noticed there was a lot of either low complexity queries, like super, super simple queries, or really, really high complexity queries with like lots of operations, lots and lots of filters that didn't make any sense. So basically I had some code that filtered those out. Okay. Um, there is a, in the more general case, there is a tool called Lilac, which kind of like helps you, uh, find more general things that you might be interested in filtering out of your data and searching your data and the, like also finding duplicates. So another part of curation is to get rid of duplicates. You don't want you know, like, okay, we did a lot of data augmentation and things like that. You might have lots of data that looks very similar or too similar, and that's not going to be good because what ends up happening is like you're going to like overweight on those examples. So like, um, there's a lot of sophisticated things you can do. You should start with dumb things if you can, obviously. So like in this case, there's 3 parts, there's 3 main parts of this dataset. There's the natural language query, there's the schema and there's the output. And so one dumb thing you can do is like to drop, uh, to drop any data where there's a, a pair that is like duplicated within those 3. If there's a pair of 2 that are duplicated, that's like one thing. And then I did— there's another, another things you can do. You can do like semantic, semantic searching and see semantic deduplication. Um, you know, that's why in Lilac, for example, you have like fuzzy concept search and things like that so that you can— and then you have like clustering and things like that. So you can kind of like look at data, try to maximize diversity, clean out things that are like too duplicate, like too much duplication. So that's kind of like an end-to-end overview. Like the idea is like this is not a linear process. I went through this in like 1 through 8, but just know that like I have to go back and forth between all these different steps and do these things differently as I hit various things. Like, you know, like I mentioned, um, I have to constantly rewrite the level 1 evals, um, you know, or I might decide to redo the level 2 evals, um, but this is, again, this is a very simple example, um, just to give you a concrete use case to give you the idea of the workflow. So that is the honeycomb use case. Let's— let me just quickly talk about debugging Axolotl. I'm going to switch gears. So like when you're using Axolotl, it's really important if you're going to use some software that you know how to debug it. And I just want to call your attention to this these docs that will show you how to debug Axolotl. But there's these guidelines here that I think are really important. So if you're going to debug Axolotl, like something is going wrong, you want to make sure that number one, using the latest version of Axolotl. You also want to eliminate concurrency as much as possible. So basically, make sure you're only using one GPU and one dataset process. Use a small dataset, use a small model. You want to minimize iteration time. And also you want to clear caches. Clearing caches is huge, like especially if you're trying to debug something about dataset formation, like, hey, it's like you don't think like your prompt is getting assembled correctly or something like that. You want to clear your cache because that can trip you up. I also have— there was a bunch of questions in the Zoom. About how do you connect the Docker container that if you want to run Axolotl in, and like, that's really connected to debugging actually in a way, like, because you can use VS Code to do that. And I have some videos and tutorials in the Axolotl docs that show you how to do that either with Docker or not using Docker and how to attach, you know, to remote hosts and things like that. Let me go back to the slides. And I already covered this. Wing, okay, so I went through, we went through a lot. I'm just gonna stop and ask you, is there anything else on your mind in terms of things like tips you might have for people using Axolotl that you you like to highlight?

Um, I don't have any off the top of my head. I— it usually comes when people ask questions that I remember, oh, you should do this, this, or this, but I don't have any off the top of my head right now.

No worries.

Um, there are a couple of— maybe this now is a good time. Um, there are a couple of questions in the Q&A, but actually some are listed as answered, but I for everyone to be able to hear them. Um, how about this one? Uh, how do you predict how long a fine-tuning job will take before you start it? Do you have any recommendations there?

That one is relatively hard to answer. Um, you know, it depends on, you know, model size, LoRA, full fine-tune, the GPUs you're using, the number of GPUs. If you're using like DeepSpeed 0.2 or 0.3 and you're having offload, it's just There's so many factors that can affect, you know, the amount of time that it takes to fine-tune a model that it's usually like— I think once you have like a gauge on a specific dataset, um, and on like certain parameters that you're going or hyperparameters that you're going to use for a specific like, you know, set of experiments, you can usually like get a good gauge on from that. But I don't have like a good like all all-around formula that works for everybody.

Yep. We're just looking through any of the other questions that— yeah, we can come back. We've got a lot of questions.

Just a second ago, talking about— someone had asked about doing a fine-tune and then improving, like doing what Hamel Husain was saying, like improving the data and then whether or not you should start from scratch again or like fine-tune over that fine-tuned model. And I think one of the things when you think about that is like, if your model is already getting pretty close to being like overfit, just fine-tuning that again for more epochs, right, is just going to definitely overfit at that point. And you should really consider just like cleaning up the original data and adding in the new improved data and then just starting from sort of starting from scratch again at that point. On base model.

Yeah, I always start again from scratch, uh, when I improve my data. I haven't thought about trying to keep going. Um, okay, I think we probably should move forward because I'm looking at time as well. Um, I think the next thing that might want to do is jump in right into Zach's.

Sure, let's do it. How do I, uh, looks like I can take over for you, so less for you to worry about. We're all seeing me all right?

Yep, perfect.

All right, hey everyone, uh, my name is Zach Mueller and we're going to be talking about scaling model training as you get more compute. How do these people wind up doing that? Uh, so a little about me. I'm the technical lead for the Hugging Face Accelerate project, and I handle a lot of the internals when it comes to the Transformers Trainer. I'm also a humongous API design geek. And before we start talking about, like, how do they go about doing this sort of what we call distributed training, let's get a general understanding of model GPU usage, right? So we were talking about how you can use things like LoRAs to reduce some of the memory overhead. But how much memory overhead do certain models actually use? We can sort of guess what that number winds up being if we're using vanilla full fine-tuning, so without using LoRAs, and then you can sort of convert some of it later. The assumptions that you basically have to have are we're going to use the Atom optimizer, and we're going to start with a batch size of 1. And for example, let's take BERT base case. So that's going to be 108 million parameters. How much GPU space am I gonna need to train that? Well, each parameter in a model is 4 bytes, and the backward pass usually takes about 2 times the model size, and the optimizer step takes about 4 times that, 1 for the model, 1 for the gradients, and 2 for the optimizer when it comes to Adam. So after doing all this computation, you wind up getting to 1.6 gigs is needed to train on a batch size of 1 for BERT. With mixed precision, that's knocked down by half because while the model is still in full precision, which I'll go over why that's important in a moment, the gradients wind up taking less because the gradients themselves are in half-bit. And so we're able to fit and roughly guess that it's probably going to take about a gig to 2 gigs overall when we're training on BERT. Now let's talk about why that matters. All right. So that's great if you have 12 to 24 gigs of GPU space, right? Typical consumer card. But what happens when we scale that up? Right. So if we look at Llama 3, 8 billion, 8 billion parameters, loading the model in is going to take you in full precision, 28 gigs. Gradients are another 28 gigs. Backward pass gets you to 56 and suddenly you're somewhere between 56 and 112 gigs of VRAM. I know I certainly don't have 56 gigs on a single card, let alone 112. If we want to avoid things like PEFT, what do we do? This is where the concept of distributed training comes in, or how do we make sure that we can use multiple GPUs to achieve what we want? So there's 3 different kinds of training when we think about it at the hardware level. So we have single GPU, right? So that's no distributed techniques. You are running it straight off of whatever GPU you have. We have the concept of distributed data parallelism, and this works by having a full model on every device, but the data is chunked and split up between every GPU. Another way to think about that is essentially we can process the data faster because we're sending chunks of our full batch across multiple GPUs to it to speed up the training time. And the last part that I'll mostly be covering in today's talk is fully sharded data parallelism, FSDP, and DeepSpeed. And these are the key areas that was sort of hinted at in the earlier discussions where essentially we can split chunks of the model and optimizer states across multiple GPUs. And what that allows is rather than having the limit of DDP where we're stuck with, say, two 4090s at 24 gigs, that's all I can use in memory, it acts as a single 48-gigabyte GPU when we think about the total RAM that we can play with to train models. And that's the secret to how you can train these larger and larger models. Now, what is fully sharded data parallelism? The general idea here is you take your model and we're going to create what's called shards of the model. So that's, say, taking the model, we could imagine a shard being it, split perfectly in half, the first half of the model and the second half of the model. And depending on how we configure FSTP, certain chunks of the training loop will happen in that VRAM space. And then depending on what points occur during that, occasionally Torch needs to know what's happening with that other model chunk because it's all the same model and we need to get the gradients all aligned. So these call what are called communications. And generally you want less of these because it's essentially time spent on your GPUs just talking to each other and trading information. You're not training anything, you're not processing data. It is quite literally just your two GPUs trading notes on how they think the model should be and then correcting themselves. Now, I'm not going to really go too much in depth into every single thing FSCP can do. What I am gonna talk about is, in my opinion, the most important ones when it comes to training in low-resource areas and when you're using FSCP and sort of how you dictate how those weights and gradients and parameters get sharded. And on top of that, I'm gonna cover some of the important ones I needed when I was doing a full fine-tune of Llama 3 8 billion without PEFT on 2490s. Spoiler alert, it was very slow. So the first part of this is what we call a sharding strategy, and the general idea here is this is us telling FSCP how we want to split all of these different things that take up, uh, VRAM. So with full shard, as it sounds like, everything's going to be split. Our optimizer state, our gradient, and our parameters. With shardgrad_opt, which is optimizer. Instead, we're just sharding the optimizer state and the gradients, and then essentially the model will be split when we're not using it and then joined back together when we are, such as during the backward pass. This reduces some of the memory overhead because we still need more than the original model, right? Because we're still fitting the entire model in VRAM, but it reduces that training VRAM a little bit for us. We have a technique called no shard, which as that sounds like, that's just gonna be distributed data parallelism. We're not sharding anything. And then the last part is a new thing that PyTorch has come out with called hybrid sharding. And it's kind of like full shard where we're fully sharding absolutely everything, including the optimizer states, gradients, and parameters. However, if you're training on multi-node, right? So multiple computers are training a big model at once, It keeps a copy of the entire model on one, on each of those nodes. That's important because remember how I said communications slow down things a lot? Hybrid shard lets us reduce the communications from, I think, 3 down to 2, if not 1. And so your training speed is increased, honestly, to some extent exponentially, depending on how long it takes for your computers to talk to each other. So the next part is we know how we're going to split the memory, right? But how do we split the model? Because we need some way to tell FSCP, all right, I have this model, how do I want to split it in between my GPUs? With Accelerate, with Axolotl, with Transformers, we use two different nomenclatures: transformer-based rep and size-based rep. Transformer, as it sounds like, is very specific to Transformers. With this, you need to declare the layer you want to split on. So like this could be a BERT layer or a Llama layer. Usually Transformers has good defaults and good helpers to help you figure out what that is. The other version is more manual and basically you're just telling FSCP after X amount of parameters, go ahead and split the model. That's great because it works out of the box. That's bad because there could be speed increases that you might be missing by having, say, like each head of a, like, Mistral model on a separate GPU so that way it can handle its own computations much faster than needing to wait to communicate with other GPUs. Now, the next part, which was particularly important for me, is the idea of offloading parameters. And what this says is, okay, I have 48 gigs of VRAM right now, if I'm assuming two 4090s, and I can't fit that. I can't train on it. Well, I'm going to accept that. I still want to do it. I don't want to go buy it through a cloud provider. And so FSDP will let us offload gradients and model parameters into RAM. Now, as that sounds like, that's going to be extremely slow, right? Because we're taking things from the GPU to the CPU and now just shoving it in RAM. But it lets us train as big a model as essentially you have available in RAM. So case in point, when I was doing a full fine-tune of Llama 3 8 billion to match a paper that came out, I wound up needing to use offload parameters because as we saw earlier, 8 billion requires about 50 gigs or so. I only have 48, and it was going to take like 72 hours to do 4 iterations through my data. Versus an hour or two on an H100. So yes, it's cool that you know how to use these tools and it can help you train things locally. Make sure to double check though, A, what your time constraint is and B, what your budget is. Because I can run it for free and it can take longer, or I can pay $5 and go finish it in an hour. Depending on how much time you have available, each solution has different, opportunities. Now, another kind of critical part, in my opinion, when it comes to doing FSDP that Accelerate and Transformers has is this idea of CPU-RAM efficient loading and also this idea of sync module states. So if you're familiar with Accelerate's big model inference, that's fine. I'll give you a brief summary. Basically, PyTorch lets us use this thing called device equals meta. And that essentially is the skeleton of your model. The weights aren't loaded. It can't really do computations too well. But it's just the skeleton for us to eventually load weights into. So rather than loading Llama 8 billion on 8 GPUs, so now we need 8 times the amount of RAM of our model to load it in at once. So that's going to be easily 100, 200 gigs, if I'm not mistaken. Instead, we send all the other versions onto that meta device so they take up no RAM, and then we load all of the weights only on one of them. And so then when we're ready to do FSDP, well, we already know we're sharding the model, so we just tell the first node to send those weights to whatever node or GPU needs that particular chunk of weights. And this really helps keep your RAM size low. And you don't suddenly sit there with crashes because, oh no, you ran out of CPU memory. Because fun fact, you will redline this quite often, I found, at least in this particular scenario. Now I've talked about FSCP a lot and I've assumed that you knew context about Axolotl, Transformers and all this stuff. Let's take it back and just focus on Accelerate, which you might not know is the foundation of a lot of your favorite libraries. So practically all of Transformers and Hugging Face as a whole relies on Accelerate. Same with Axolotl, FastAI, anything Lucidrain does at this point, as well as Cornea. And the general idea with Accelerate is it's essentially 3 frameworks. You have a command line interface that Hamel Husain and Wing already showed us whenever they were doing accelerate-launch. You have a training library, which is under the hood what is doing all of this distributed training fairly easily. And then the big model inference that I mentioned a moment ago. For the sake of this talk, we're not talking about big model inference. We don't particularly care about that here. We're just caring about fine-tuning LLMs. So we're gonna focus on the first two. So you need about 3 commands to really get everything going. The first is accelerate-config. This is used to configure the environment. This is also what Wing has managed to wrap around beautifully when he shows his Accelerate launch commands, because his config files can directly be used for doing Accelerate launch, which is phenomenal. The second part is estimate memory, which goes through those calculations I showed a moment ago whenever I was playing around with the idea of, well, how much VRAM can I use? And the last part is accelerate launch, which is how you run your script. Let's look at sort of why these matter. Launching in distributed training sucks. There's a lot of different ways you can do it. There's a lot of different commands you can run. Some of it's PyTorch, some of it's DeepSpeed, and all of them have slightly different commands, right? So here, if you just do python script.py, it's not gonna train in any distributed scenario. Most will get model parallelism, but you won't get like distributed data parallelism. FSTP don't work, won't work. torchrun and deepspeed are the main two commands you can use to run. This will basically say torchrun, run on a single computer with two GPUs, my script. And then it does some things in the background to help make sure that works. And that's a lot of different commands that you have to know and remember. And so accelerate launch is here to just say, okay, tell me what you're doing and I'll make sure that we're running it. So for— it operates by these config files similar to what, again, Wing was showing us with Axolotl. And these essentially define how we want certain things to run. So here we're saying I have a local machine that's multi-GPU running with BF16 mixed precision on 8 GPUs. With FSDP, on the other hand, we can go through and specify everything we want to use with FSDP using a config. And this way, Accelerate launch just knows, hey, we're going to make sure that we train in FSDP if we're using Accelerate. And that's all you need to do from a launching perspective. And if you're using Axolotl or Transformers, this is all you need to do. The next part I'm going to show is sort of the internals of it on the low level of how Accelerate works and how you can use Accelerate specifically. But do you remember this isn't necessarily needed if you're using things like Axolotl or Transformers. So the general idea with Accelerate is we want a low-level way to make sure that this can essentially be device agnostic and compute agnostic, right? So make sure you have your code running on a Mac, running on a Windows machine, running on a GPU, running on CPU, running on TPUs. And it does so in a minimally intrusive, and ideally not very complex manner. You create an accelerator and you just have it prepare all your things. And that's it. You're off to the races. Switch your accelerator or switch your backwards function to use accelerator.backwards. And on a whole, that's most of what you need to do. How it winds up working is similar to FSDP. Accelerate will do the data sharding for you in taking your data and splitting it across GPUs. It also operates by essentially having one global step. So an easy way to think about it is if we're training on 8 GPUs versus a single GPU. So if a single GPU had a batch size of 16, and now we're training on 8 GPUs, the equivalent in Accelerate to get the same exact training would have each GPU have a batch size of 2 because 2 times 8 is 16. And so what winds up happening is this lets us successfully scale our training with— that should have roughly the same results when training on a single GPU versus training on multiple GPUs without needing to worry about, oh, do I need to step my scheduler more? Oh, do I need to adjust my learning rate more? Oh, do I need to do this? Do I need to do that? You do that, it's the same amount of data being processed at one time and everything else is just done for you. Now, the next part of this, I want to talk about some very specific tweaks that we do to protect you from dumb decisions. The first part is mixed precision. This is a bit different than maybe your normal idea of mixed precision. We don't convert the model weights to BF16 and FP16 when we're training with Accelerate, and we try our hardest to make sure that doesn't happen. Instead, we wrap the forward pass with AutoCast instead to just convert the gradients. This preserves the original precision of our weights and leads to stable training and better fine-tuning later on. Because, and this is very important, if you go to BF16, you are stuck in BF16. There was a whole issue a few months ago with Transformers where some quality of some fine-tuned models weren't doing well. This was the cause. Now going a bit more than that, if you're familiar with, uh, or keeping up to date with efficient memory training, you might have heard of something called Transformers Engine or MSAM. Uh, the idea behind this is we make use of like 4090s, H100s, and do training in 8-bit. Now this is different than quantization. You are actually training on raw native 8-bit. So 8 bits and that's all you have. A lot of mistakes I see people do with this, especially with the NVIDIA examples, is they do the prior thing of converting the entire model into BF16 and then train. That leads to huge instabilities during training and generally people's performance hasn't been the best. I've also heard rumors though that even this can go bad. So it's always worth playing around with, if you have the ability, FP16 versus non-FP16, and that includes BF16, and testing out sort of what levels can be in 8-bit. Because like with Transformers Engine, it's still using the AutoCast. And so the computations, rather than being done in 16-bit, are done in 8-bit. And then if you're playing around with MSAMP, that lets you experimentally go even further with this. And so it can, you know, we can get to a point where if we do 0.3, almost everything is in 8-bit. Your master weights are in 16-bit and your optimizer states are even in 8-bit. I'm scared to play around with that. I don't know necessarily how good that is. I need to play around with it. And that's sort of what I'm using the Llama 3 training for, to just to play around with these things. But it opens up opportunities if you have the compute to do this. Now, the last part I'm gonna very briefly talk about, and we can talk about this more in my office hours, is DeepSpeed by Microsoft and fully sharded data parallelism. These two are almost the exact same. DeepSpeed has a few tweaks and calls things a bit differently. But if you've done it in BF6 or in FSDP, it can be done in DeepSpeed and vice versa. A wonderful community member recently posted some documentation where he directly talked about this parameter in DeepSpeed is this parameter in FSDP. And generally what I've seen, it's a mix of if people prefer DeepSpeed or FSDP. It's usually a matter of do you want to go with Microsoft and do their thing or stick with PyTorch and just stay native? But either can be used interchangeably as long as you're careful about setting up the config. So as a whole, uh, XLR8 helps you scale out training, especially with using FSDP and DeepSpeed, uh, to train these big models across a number of GPUs. You can use techniques like FP8 to potentially speed up training and reduce some of the computational overhead. But when using mixed precision in general, especially with FP8, be very careful about how you're doing it, uh, because you could potentially lock yourself into that weight for you and everyone else. So I'll post this presentation, of course, in the Discord, but there's some handy links there that will help get you started with Accelerate, go through some concept guides to understand some of the internals and really get you going. So yeah, there we go. Let's look at some questions. Let's see. I have one here. I thought that DeepSpeed 03 is the same as FSCP, but the other options in DeepSpeed weren't necessarily equivalent. Uh, it's gotten to a point where there's some equivalencies now. Uh, the chart talks about it. Uh, 0.3 is definitely the equivalent of FSDP. Uh, but there's some tweaks that you can do because FSDP gives you options to only offload certain things.

I just want to mention that, okay, I didn't show you, there's a DeepSpeed and FSDP configs. Like when you want to do multi-GPU training in Axolotl, in— you have to supply a config file. I'll show you some examples, uh, of those. Um, they're in the— I can— whenever Zach's done, I'll share my screen.

Yep, sorry, there you go.

Okay, I'll just do it right now. Uh, let me find—

yeah, I have some clarification while we're pulling that up.

Yeah.

Um, so one of the things, especially for the FSDP part in the Axolotl configs, is we try and move those FSDP-specific configs into the axolotl and then it like maps them into Accelerate. What we found was that a lot of people were running accelerate-config and then setting up like setting things and then they would go and use axolotl and they would have like a mismatch in certain parameters. And what would happen was it just would break in most, in a lot of situations. So what we actually recommended people do, we added warnings saying just remove your accelerate-config and then we will sort of map all of those configurations that normally get set by Accelerate through like, I think Accelerate uses like environment variables to sort of communicate that under the hood anyways when you use Accelerate launch. So we just sort of like mimic a lot of that just to like avoid some of the headache of doing it one launch, running Accelerate config, getting a mismatch later on that just cause a lot of support issues. So, um, that's just something—

makes perfect sense. That's exactly the solution I recommend. Like, even I'm debating on rewriting half of our internals for the FSDP and DeepSpeed plugin because, like, I don't necessarily want to rely on environment variables. And even setting it up, I'm sure as you've experienced, normally is problematic at best. So, uh, yeah, that's a very smart way to go about it because it's— even we've had users that report issues and like, well, it's because you set up your config wrong and you're using something else.

Yeah, I mean, and so that's like what you heard from Zach today about like stage 1 to 3, BF16, all that. That's all like background that you might want to know to like demystify a little bit about what is happening when you supply these configs. What I do honestly is I just use a config again. I just use one of these like 0, 1, 2, 3, um, you know, or the BF16 one, and use it, kind of use it off the shelf, and then maybe consult— like, Zach has like written a lot about this. I actually look at his presentation. He's given like similar versions of this before and posted online. He will today post his slides, and I kind of fiddle with it a bit sometimes. But honestly, I just use ones that work. If I want to parallelize my model, especially using a bigger model and parallelize it across GPUs, uh, then, then I'll, I'll pick the right config. and you specify, like, you have these configs in the axolotl repo, uh, and then you supply it to the config, the main config. I'll show you an example when we talk about modal in a second.

Can I, can I add a clarification on this one specifically?

Yeah.

With 0.1 and 0.2 specifically for DeepSpeed, um, you, um, I think the BF16 and FP16 can be set to auto because it doesn't, DeepSpeed doesn't care about it until after the trainer is loaded. But for 0.3 specifically, and I see Zach nodding his head, is it needs to know ahead of time specifically that you're using BF16. So you actually have to— you can't set auto in the 0.3 config if you want to use BF16. So that's why it's set as like there's a specific 0.3 BF16 because it needs to know that you want to load it in BF16 before it ever— before it before the trainer sees it or something along those lines. Maybe Zach can explain it better than I can, but—

No, that's a pretty good explanation of it. It's something with DeepSpeed when it comes to setting up the actual call to DeepSpeed and initializing everything. It has to know well beforehand what we're actually doing, which makes it a little annoying whenever we're dealing with configs that way.

Um, okay, I think, uh, we should probably move on to the next thing, which is training on Modal. Or Zach, just want to make sure you're done with—

yep, you're good.

All right, um, so there's a lot of different ways you can train models. There's— you can use RunPod, which Dan showed earlier. That was like done on RunPod. That was the like recording. If you look at the Axolotl docs, actually, it'll show you, it'll tell you a bit about runpod. If you just search from runpod here, you'll find a little bit there. But also there's a Docker container for Axolotl, which is like what you want to use most of the time. Um, Wing, do you want to say anything about that? Like, what's your preferred way of running? How do you run it stuff? Like, what's your compute.

So on my local 3090s, I, it's, I don't use Docker containers just mostly because it's like development and it's just not amenable to using Docker containers for that. Um, but for general like debugging issues that people are seeing, um, I will just generally just spin up a Docker container on my RunPod and debug the issue there. So because it's just environment. It doesn't have all of the mess and mismatch of like various, um, packages that I might not have updated.

Makes sense. Um, and then yeah, if you look at the README, there's a whole bunch of stuff there about it. Um, okay, so Modal. What the hell is Modal? So actually, so okay, like just some general rule about this conference. Um, we were pretty selective about the tools that we brought in to this conference, or that I'm gonna talk about. I'm only gonna talk about tools that I use or that I like. There's like hundreds of tools. Um, one, and you know, one that I really like is Modal. So like, what is Modal? Modal's actually like this really cool cloud, uh, native way to run Python code. And the thing that's really interesting about it is like it has this, uh, like one innovation is like it feels like local development, but it's actually remote development. There's nothing to do with fine-tuning right now. Just, I'm just telling you a little bit about models. He has some background. And basically it's also like massively parallel. You can, you can get, so like things like Axolotl, it can do easily do like fine-tuning. Actually like Wing, how do you do, how do you do like hyperparameter search with your Axolotl training? Like, what do you like to do?

It's manual right now. It's like, okay, so I change out the learning rate.

But yeah, it makes sense. Um, so like, a lot of times I do, uh, use something like Modal, or I'll use Modal to do things like hyperparameter tuning. There's different ways to do hyperparameter tuning. It's not something you should focus on like in the beginning, and it's totally fine to do it manual. I do a lot of things manually. I use bash scripts. Sometimes, uh, to do like many different Axolotl runs. So, um, it's very like Python native. There's these, uh, Modal docs which are here. If you're just getting started in Modal, actually, like, to really experience this like magic of Modal where what I'm— you're like, what am I talking about, this like local but it's remote? Like, what does that even mean? I don't even know how to explain it to you without you like trying it yourself. So like, this is like— I— so there's a lot of docs here in like Modal. You can go through like the hello getting started one, but I actually think like what I like to show people first is this like web endpoint one. I'm not going to demo it right now because I don't have time, but basically just like try it out. And basically what you want to do is like you can change the code and you can see it change in production in real time, and you don't have to do these like deploys, like constant deploys to change code. It's like this really iterative, like, interesting thing. And I've built, like, lots of tools in Modal. I have built, like, this transcript meeting transcript summarizer with Modal, uh, also Weights and Biases webhooks. Uh, the links are— that are going to be in the slides, so I won't labor that too much. Uh, the one thing about— so for Modal, uh, for Axolotl, they have this repo called LLM Fine-Tuning And it's a little bit different than— it's like wraps Axolotl. So that, that's interesting. Like, Axolotl is already wrapping so much, why we need to wrap Axolotl? Well, actually, like, um, it's kind of interesting. Like, if you have a workflow that you really like, um, you might want to abstract it a little bit more. And plus, you can get all the benefits of modal by doing that. Um, certain things you might want to know about this repo is, um, when you run the train, it automatically merges the LoRA back into the base model for you, um, by default. You can turn it off. And then also, like, one key thing is there's a data flag you have to pass. You can't rely on the, the data set in the config file. You have to pass a data flag, um, and then the DeepSpeed config comes from the Axolotl repo itself. So you have to reference sort of like the Axolotl repo, what I was showing earlier. It's kind of like these are mounted into the environment, this deepspeedconfigs. So it's kind of like a beginner's way of using sort of Axolotl with Modal, but it is, um, it's something to try first. And like, it's kind of like you can tweak it, you can tweak it, you could change the code. Um, but basically, like, you know, there's the README here. There's a way to get started. Obviously, you have to, you know, start Modal, install it, and essentially, like, what you do is you clone this repo and then you launch this fine-tuning job. And basically, like, this command, um, the detach thing just makes it run in the back— like, makes it run in the background so where you can do other things. Um, but there's this, uh, there's— here's the entry point. This is basically where we're wrapping the Axolotl CLI command in this train function, and then you pass in the config file and then the data. Okay, so it's like very similar to running Axolotl, just wrapping Axolotl. I'm going to do a really quick video of what that looks like here. So, um, you know, just do modal run, and then basically, you know, it will go ahead and, and do your Axolotl run if you want. And this is like running the exact example in the repo, um, and you can do the same things. You can put your Weights and Biases and your Hugging Face token and so on and so forth. So let me go back to the example. Oh, sorry. Let me go back to the repo. Sorry. And just to point out here, just to navigate yourself in the repo, there's this— actually, I'm going to hit the period on my keyboard to show you VS Code real quick so I can just show you some code. And so the source code, like the code for Modal, is in this source folder. And the training part is maybe what you want to take a look at if you're curious on like what is happening. And the entry point that we demoed right now is this train function. So there'll be a train function here in— there'll be, you know, in this file right here. Let's see. And then the common.py, that's actually the setup. Okay, that sets up the environment, that sets up the Docker container and installs some dependencies and makes your secrets come in. You don't have to worry about this. I wouldn't actually look at this like in the beginning. I'm just showing you around so that if you wanted to dig in, you could check it out. I think it's pretty cool. Um, and then one thing I want to point out is like there's these config files. If you want to run the demo in the README out of the box, there's this like very small, uh, training run that basically overfits on purpose. Um, you just have to know that, okay, the dataset here, this is just— this will get replaced by whatever the data flag that you pass in. Um, and then you just know that like, okay, uh, for this DeepSpeed is actually being used here. So, uh, that's what we just talked about. That was the background that Zach gave. And this is actually being mounted from the Axolotl repo because remember the Axolotl repo has this DeepSpeed configs and this is being used. Um, so just so this is Just orienting you to that. And let's go back to the slides.

Whoops.

How do I go to the next slide? Another thing you might want to do is debug the data. So like you can run an end-to-end, but remember I told you, you don't want to do that. You don't want to just train stuff. So if you want to do your, have your own data inside Modal, I have this notebook here. So let's go to this notebook. Whoops. Let me just go to the repo and go back and go to the notebook. So I have this notebook here about inspecting data. Okay. And I'm just going to change this GitHub to nb-sanity. Because it's easier to read. Um, and basically, uh, this— you kind of do the same thing, is like, you know, and just make sure this is a way that you can inspect the data. So you can do model run but then pass a preproc only flag. And what happens is the logs will print out a run tag, and with that run tag you can see the last run prepared folder, essentially. And like the last run prepared folder, you can just get that data and analyze it the exact same way that I showed you in the Honeycomb example, essentially, which is like, you know, and then print out just to make sure the data is in the right format. So I think that's important. You might want to do that if you're using this, and just this is a notebook that might help you. Okay, I think that's it. And yeah, we can do Q&A.

Okay, let's talk about— I will MC Q&A. We have some questions that were answered, typed, but just so that people hear the answer, I'm gonna do a mix of open questions and answered questions. A couple of, in case there are common questions, will office hours be recorded? The answer there is yes. Are tiny models like Phi-3 more or less suited for fine-tuning? You answered that in text, but for others to hear it since it was highly voted, You want to tackle that, Hamel Husain, or anyone else?

I usually don't go smaller than a 7 billion parameter model because I haven't had to go smaller than that. Like, that's like a really sweet spot for me because the models are like kind of good enough and they're small enough. But I don't know, Wing or anyone else, do you have any opinions on this or seen anything?

I haven't spent a lot of time with the Phi-3 models, mostly because I wasn't impressed by, I guess, the Phi-1 models. I feel like they were just way too small. And, um, there's— I think with the smaller models, just the reasoning is worse. So I just— Llama 3 is good enough and it works. So yeah, 7 billion.

But how to determine the adapter rank? There are actually two param— this wasn't part of the question, but there are two parameters that go together. There's the adapter rank and then the adapter alpha. Um, someone said How to determine the adapter rank? What do you guys have to have for that one?

I just copy the config, so I don't determine anything. Bing determines it for me.

That's one of those hyperparameters you should play with, assuming you have good evaluations, and to just understand, is your model— is that LoRA at that rank sufficient to get good accuracy on what your downstream use case is? So 32, 16, 32 is like typically a good starting point that you see most people use. And then so for rank, it's— and then for alpha is usually, I believe the paper says it should be 2x the rank, 2x the rank. And then if you're using something like, I think it was like RS-LoRA, it has something to do with the square root, but I try not to get into that.

There's a blog post. I'm forgetting, I think by Sebastian Raschka, where he actually has a— does a grid search and talks about, um, what works for those. So I'll try and share that with the community. Um, yeah, yeah.

There's another thing that I do, and this is kind of a weird answer. Um, I actually ask my friends, um, who are a lot smarter than me. So there's this guy John O'Whittaker. He's— he like really understands a lot of stuff. I'm like, hey, what rank do you think I should use for this? And he gives me some tips. John O is actually speaking in this conference. He might not talk exactly about this, but he has a really cool talk called Napkin Math for Fine-Tuning, which you should check out.

Okay, I'm going to switch over to some open questions. I'll take the one that's set up top. I have a custom evaluation or benchmark for my model. Is there a way I can get it to run periodically during fine-tuning to see how the training is going so far against that evaluation metric? It's actually something that I've wanted. I don't know the answer to it, but it's something that I've wanted in the past. Wing, I think that's, since I just read it, and does that question make sense to you? Do you understand the question?

No. Basically, can you have like an evaluation function in Axolotl or something, some callback or something like if you want to, compute some like custom evaluation metrics. Like, how do you deal with— do you do that? Do you— how you deal with it?

Like, there, there's, there's like the TinyBenchmarks that you can run sort of against sort of the more standard benchmarks. Um, as far as trying to get more like custom evaluations, it's not really supported right now. I think you could do things by adding callbacks on the evaluation loop maybe and doing some janky pulling from disk, things you wanted to— I guess so. Here's something you could probably try. There is a way, I think on the evaluation, if you were to specify a custom test dataset for for your evaluations, you can have it generate predictions for those at certain steps and then log those out to Weights and Biases. And then you could, like, pull those from Weights and Biases and then do your own, like, evaluations using, like, LM as a judge or something along those lines. That would be one way you could do it. But there's nothing, like, directly integrated right now that's sort of streamlined for that.

How would you do that dumping of predictions in Axolotl? Like, how would you do that?

Yeah, so it's already built in. I think it's like the— there's something called the eval table something setting in Axolotl. What it does is it will pull some number of prompts from your test dataset and then run predictions during the evaluation step and then log those out to log those to Weights and Biases. I think it's like eval_table something. It's a little, it's, it's a little bit flaky, so it's not like a top-level thing that I've used. I think there was a contributor who submitted that. Yeah, eval_table_size and eval_max. So I believe the table size is the number of Yeah, the number of predictions that you want to do, and then the number of max tokens is how long you want to, like, how many tokens you would like it to generate during that eval step.

That makes sense.

Good question.

I like this one. Given Axolotl is a wrapper for some Hugging Face libraries, are there any important edge cases of functionality that you can do in the lower-level libraries that aren't yet possible in Axolotl?

I'm sure there are a lot of things that you could do.

Um, there's tons. Yeah.

There's tons.

Cause then you're operating at the code level.

Yeah. It's hard for me to keep up with everything else that goes on underneath.

So like, yeah.

Yeah.

You can have custom callbacks and stuff. You can do this eval thing that we were just talking about. You know, you can do all kinds of stuff. Yeah.

I think it would especially be like at the speed that Wing can implement whatever we chuck into Accelerate and more specifically. We can then chuck into the trainer, and it's whatever that gap is, is the bleeding edge that you don't have access to, you know. And so like, that could be like new FSDP techniques, new DeepSpeed techniques that get added that we need to update in Accelerate and then push to the trainer. That, I think, for the most part, should be the most major gap, because we try and shove everything we can in Accelerate into the trainer that then Wing gets for free.

But I think this, um, flexibility for callbacks during training with whatever you want to do, like at each batch or at whatever frequency to calculate custom evaluation metrics or stuff your data who knows where. That would be like the sort of thing. There aren't a ton of use cases for that, but doing stuff in between batches seems like a— these sort of callbacks seems like an example.

Yeah, but you might be wondering like, okay, if you— why use Axolotl? It's worth bringing that up again. I just want to like show you like one example is like because there's a lot of stuff that you need to glue together, especially if you don't have a lot of GPUs. So like one example that came out recently is like, you know, QLoRA working with FSDP for the longest time didn't work. And the Answer AI team kind of enabled that. And then within hours, Wing like glued it into Axolotl, like really before anyone else. And so I was able to use it like almost right away. And Wing keeps doing that, like over and over again for like anything that happens, like the, like, you know, the LLM space is like changing extremely fast. Like from day to day, there's like a new technique for like efficient fine-tuning, like lower GPU memory, faster, whatever. Something like the ones that are really important, like, like this one, get into Axolotl like really fast. And trying to do all that yourself would take a long time.

There's a question. What are the practical implications of 4-bit versus higher precision? I think we said that some of those, um, we will talk about more at deployment. Um, is there anything that you guys think we missed in talking about the implications of, uh, so 4-bit's obviously, um, going to lead to a smaller LoRa and requires, um, less RAM. Anything else?

You know, 4-bit is quite— I mean, it's pretty, it can, you know, it can be aggressive. Like I have noticed the performance degradation when going all the way to 4-bit before. Like I've been using this library MLC, for example, and they have like 4-bit quantization. And, you know, in that I did see a difference. I don't see much of a difference 10 to an 8-bit, but I'm just talking about vibe checks. There's probably like papers out there that do some analysis. You always have to check yourself. It is worth like just doing it and checking to see like, and running your evals to see what happens. Um, but generally like the trade-off is, okay, you, you know, like for the smaller models, uh, you know, you'll have a more portable model that's probably faster, probably, uh, you know, maybe now it fits on one GPU. You don't have do distributed inference, things like that potentially. Um, but then it might come at a performance hit, so you have to like do your evals to see what that performance hit is.

Yeah. Um, and one thing to keep in mind is QLoRA is definitely like a trade-off when you don't have enough GPU RAM. So if you're training, if you have an H100 and you're training like a 13 billion parameter model and it fits like don't decide to go down the QLOR because you lose a lot of performance in the quantization, dequantization step. And like I experimented when like QLOR came out, I was like, why is this really terrible on A100s? And like, it should be faster, right? No, it's like, it's because of the like quantization, dequantization steps that it's just actually worse when you're, if you're going for like speed and performance. When you don't actually need it. So it might be an over-optimization in some cases.

It's definitely a GPU-poor optimization for sure, which is like lots of people.

Yeah.

Does Axolotl also support, um, Mac M-series GPUs?

So yes, um, because, um, so PyTorch is supported on Mac M-series. Like, there is like an example somewhere where someone did, uh, did it, but you're probably better off using like MLX, I believe is the repository that does like, has better fine-tuning for like if you want to fine-tune on your like your MacBook or what have you. Um, I think, yeah, I think it's called MLX, right?

Yeah, yeah, it's MLX because yeah, fine-tuning on Macs is 3 different frameworks, 3 different backends, and all of them kind of work. So, um, it can work. Your mileage may vary.

We got a request for your slides, Zach. Can you— I assume you'll be able to share them with, uh, yeah, with everyone.

Uh, they're actually already in the Discord.

Great.

We can probably upload those as well along with our slides.

Right?

Yeah.

Yeah.

Yeah. It's just a web URL, honestly, because mine's actually hosted on the Hugging Face Hub.

Oh, fancy.

So in an overarching sense, are there mental models or intuitions that we bring to agentic LLM applications versus ones that are not agentic?

Yeah. I saw this question. Mental models we bring to agentic versus non-agentic. I guess like in a sense, okay, like, okay, what is, what does agent, agentic means? Agentic is like some workflow where there's a function call. Um, really it's like models that make function calls are quote agentic. I just want to demystify the terminology. People just like have terms and then feel like it's rocket science. I actually have not worked on a use case where there isn't some function call involved. Like even the Honeycomb example, like, it's, you know, it's executing a query at the end for you. You know, that's after like the query generation, but it is executing it. And it's going in some loop, like after that to try to correct if something goes wrong. Um, and so like, and really everything, you know, it's really hard to think of. I mean, there might be some use cases that, you know, but there is no function calls, but I feel like they, they all that I've had had function calls. I think like you need to write evals that you can kind of think of it as like unit tests and integration tests. Like it's important to, you know, have tests that test the function calls. And have a unit test for those as well as like integration tests. That's what I would say about it.

All right, actually I got, I got one. Is fine-tuning an LLM to output deterministic results exactly the same? So this is, I think, important because, um, to output deterministic results is not something about how you do training. It is instead something about how you do inference. So you're going to train the model, it's going to have some weights, And then when you are predicting the next word, the last layer is this softmax so that the output of the model is actually a probability distribution over the next token. And then to make that deterministic, you would just choose the whatever word is most like or whatever token is most likely. But that's all something that is— and then if you don't do that, you're just sort of sampling from this probability distribution. That's all something that happens at inference time rather than something that happens at training time.

I'll give you a little bit more nuance there. Is like, um, if you— okay, if you want structured output from your LLMs, uh, this guide— the guided generation that Dan is talking about is like you can clamp down the model so that it's only providing you only tokens that make sense for like in your constraint. So like if you want a JSON output with a certain schema that only has like allowed values, you can have a grammar or you can write— it's like basically rules that clamp down on the model and like on what tokens it's allowed to predict. Um, fine-tuning can, you know, if you have like a very specific type of structured output that you want the model to always provide, um You know, so like basically, like, you know, fine-tuning can make it happen more reliably. You know, the— it's like a trade-off, I guess. Like, you know, if you're doing fine-tuning correctly, you should, you know, hopefully you don't trigger the guided generation framework that often. If your guided generation framework is getting triggered very often, then, you know, perhaps that means that if you're already doing fine-tuning anyways, perhaps it means that your fine-tune is not that good. But the cost of the guided generation isn't that— isn't very meaningful. The guided generation frameworks are actually like really good and really fast. Like, you know, things like Outlines and things like that tend to be really good. But it turns out that fine-tuning can help quite a bit in like learning syntax, learning structure, and things like that with more deterministic outputs.