Axolotl Office Hours

I see the first question is about template-free prompt construction, which is like my favorite subject. Remember when I wanted this because I didn't want a template?

Where it's just input output?

Yeah, just input output and then you have the label.

Yeah. Yeah, so that one really is just whether, you know, if you've sort of like landed on you know what your chat template looks like and you're comfortable like converting it into your chat template and knowing like these are the inputs, these are the outputs, these are the things that I like in this example here, like label is true. So those are things that maybe those are things that count towards loss. And maybe in that example, that first one, that hello might have— should probably be— we should change that doc file to be false. That's like the, you know, you're— that's like the user talking to the bot. And then sort of like, so that would be labeled false because you don't want the model to learn how to just, you know, introduce itself, I guess, in this particular context. And then sort of label true, you're asking the model to respond as, hi there. In this example, maybe we should put like a, like a, the, the /s, the, you know, the EOS token just so we know to stop inferencing there.

But generally, like, these are good for things like multi-chat, multi-turn chat. And I think there's a lot—

like, this is probably easier to grok for some people because it's like, oh, this is— it's like, I understand what's the input, I understand the output, and there's none of these, like, weird templating things that you have to sort of go and figure out. It does, like, link lean itself to a little bit less flexibility because like if you wanted to like change chat templates down the road, you'd have to go find your original data format and then like just reformat it into this particular format. But I think for some people, like once you've just landed on something, this can be like very valuable. And I know that I believe like some platforms, you know, it's very common for some platforms as well where you've already got input output turns, like it's just the input data and the output data. Um, so it's like it translates very well to like the sort of the old OpenAI, I think, format where they had inputs and outputs as well.

Yeah, and like, um, concretely, like one reason I wanted this for myself was, you know, I didn't want to be locked into the kind of the chat template because I had like different kinds of roles that I wanted, like special roles or whatever. I wanted to make up my own roles. Some of them should be masked, some of them shouldn't be masked, whatever. And I just wanted like to just work directly with the strings, if that makes sense. So this is a little bit advanced. Like, I wouldn't— I would not recommend that you start here. You should only be looking at this page if you find that that is like this, that one of the chat templates don't work for you. And but, um, but if you're just starting Learn Axolotl, like, do not look at this page. It's very advanced.

And then just to follow on up, it seems to me— and you correct me if I'm wrong, and I haven't used this— um, if you are using this and then you're doing inference, you probably along the way spend a lot of time just looking at the strings to make sure that you— you're taking a lot of responsibility for creating the strings correctly that you're going to do at inference time.

Is that right?

Yeah, yeah, you're taking more responsibility. You should always take lots of responsibility with getting the strings right, but then you have to be— yeah, you just have to know what you're doing and have it be the same at inference time, for sure.

Yeah, I'm going to mark that question as, uh, answered live.

Yeah, another like nuance of this is like sometimes I don't want a chat interface, but I do want input and output tokens, or I want to mask. Like the Honeycomb example, there's a question the user asks and then there's an output, but I don't want it to be chat because in the application layer it's not really chat, just question goes in and query comes out. So in that case You know, in, in like the real use case, like not beyond the toy demo I showed in the class, like, yeah, I used this template-free approach.

Okay, I sorted by upvotes, but I think, Wing, you're, you're picking them out. So you want to pick a next question?

Uh, decide data type, uh, dataset. I want to see how to set the side data type for datasets.

On Hugging Face.

All right, so let's see.

Let me share— let me look it up. Hugging Face. All right. Sorry. So share screen.

So you guys can see this, I'm assuming.

So let's take a look at like, this one might be a, let's start with like the sort of like more like what started came out last year, right? Alpaca, right?

We look at Alpaca datasets.

I like to use this one.

Typically what this is is like, it's the instruction.

So this is like, all right, here's sort of the prompt. The input is like, I usually like to say this is like a follow-on to the instruction. So like, identify the odd one out and then you're saying, all right, and these are sort of like the inputs to this instruction.

A lot of times these can get—

Sorry, Wayne, can you just zoom in one time?

It's a little—

text is a little small.

Yeah.

Oh, yeah, that's— I think that's better.

All right.

And then sort of the output, right? So, like, if you see the text here, this is usually, like, a concatenation of, like, all of these fields as well as, like, what appears to be the system prompt. So, like, if you were to use this one, you could use, like, a completion—

what—

like, the completion style where it's just like takes everything from the text field and it just like uses all of this. The downside of using completion in text here for an instruction-style dataset is it's like the loss is calculated across every token, like whether or not, whether or not like given below is an instruction, whether or not the model itself predicts like the word that describes a task, which typically you're not trying to condition the model on, right? You're really trying to condition the model only on the output here. So that's like Alpaca. Like, if you look at the traditional— I think the first ones that came out last year was like SharedGPT. This was sort of like a leak, I believe, of like the original SharedGPT dataset or something that a lot of the SharedGPT models were using. Actually, their dataset viewer is not showing up. So let's see if we can find something. Where ChatGPT, maybe this one, ChatGPT, yeah, ChatGPT style.

So from this one you have like, so there are a couple of variations on this.

Sometimes you'll see human, like you'll see human here or user for the, like, for the user turn, and then you'll see, like, GPT or assistant here for the, for the model turn. And then there's variations where you can have from, like, from human, or you can have the role as human, and then you also see variations where there's value here, or there is content. If you just use the type ShareGPT, it does, it will, if I remember correctly, I believe that one will try and look at the, at the, at the row and determine if it should use from— like, it will automatically figure out, like, several of those variations and then just format the data correctly, figure out, all right, I need it. It'll import it into the correct message list, and then downstream it'll do all the rest of it properly.

So the general Shared GPT is pretty, it's pretty well like flexible for most use cases because like, so this one is from human.

I think if you look at, let's say, OpenHermes, which is also like a Shared GPT style dataset, it is, this one's from human. Value. There's some other ones. But, you know, if you look at the— if you go and look at the code, Source Axolotl strategy, right?

Let's see.

You can see like here, get conversation thread. It'll try and like determine if it's a role, like it'll look at the keys and like do a lot of that for you, right? So I wouldn't be too concerned about that.

And then sort of going, I think there was a similar question somewhere else about DPO and RL configurations.

So in the same vein, I'll go ahead and talk about that. So like in the DPO, we have like various like a lot of conversions of—

and mostly I name these for—

because there's not like any standard naming. So I'm just like, all right, what are sort of like the datasets I'm seeing? And just sort of like, so if we look at like Argila Chat, right? So if we look at like this dataset here, we will, you know, that isn't it.

DPMIX.

Oh, they must have renamed this one.

Oh, DPMIX, right?

7K, right? That is this one. Yeah, so that is this one. So you can see that you sort of have to look at the dataset, see, all right, they have like chosen, rejected, and they don't have like an initial prompt, right? So this one, this one compose will build the prompt based on everything up to the last response. So the prompt is—

so the prompt between both of these should be that everything up to the user role is the same, I believe.

Question.

I think everything in the first user role is the same.

And then sort of the content is different between the chosen and the rejected. And this is also one example here where we have content instead of value, and we have role instead of from, and then user instead of human, right? So there's a lot of variance in sort of like these ChatGPT-style datasets.

If we go back and look at like the Intel Orca DPO pairs, so those are like, I think those are some Intel or DPO pairs.

You can see here they have like the question, the chosen, rejected. These are like much, much simpler to understand, right? It's like, here's, here's the prompt in this column. Here's the chosen response and here's the rejected response. And then if we go and look like, I think someone recently like added support for like KTO and, you know, did a lot of these to map these to KTO for KTO data sets, mostly because KTO, you don't have, you don't have, you don't explicitly have rejected and rejected and chosen for any given prompt. You have like this prompt and this response and whether it was rejected or, rejected or chosen. And then because KTO can sort of like doesn't have like preference pairs necessarily.

And then sort of let's say user-defined, that goes back to— so there's also like, that's also in the documentation probably where if you I don't remember.

Have you used both the KTO, DPO? Just, I'm just curious.

I, so I usually do the DPO. So a contributor recently added the KTO, like, I think just this past week.

So it just got merged in this week. So support for that. So I haven't had a chance personally to use that. I believe it's not template-free. Is it template-free?

No, there is a— there's like user-defined datasets.

I know that Zach has been using that a lot, some experiments where you can like just— let's see if we can find a good example.

The modal one may be using like a more fine one because it's like I see the string templating in the YAML file.

Yeah, let's see, where's config?

Config, do the Memrise. Yeah, even that one maybe too.

Yeah, yeah.

So they use like sort of these where you can define the formatting in the YAML file. You can actually point to, you know, you can define what field to pull in the instruction from and the input from and the output from. I mean, just be aware that the output isn't—

it's sort of like implied just because, you know, it's—

so really all of this is like the user input here for the format, and it's implied that sort of the model output is the output field, and there's like some sort of EOS token or end of string token that's appended to that as well automatically. So this can be like These are okay for like some use cases, but when you start to get into more advanced use cases where like with Llama 3 having like an end of turn ID, which is different from the end of sequence ID, it can get a little hairy. Maybe it's worth rethinking how this is done, but this is like if you don't feel like— if you have a specific use case where it doesn't fit into an existing chat template or it's sort of you have some, or you have some sort of custom template that you want to use, this is like, and you don't want to write like Python code to handle all of this, this is like a pretty reasonable way to do it.

Now what happened, so we were really paranoid about this in this example because of the space. We're like, does it, because we're like, okay, I know the output gets appended to the end of this. Like sometimes people like would add a space. We're like, oh, do we need to add a space to the end of the in the last closing bracket in the INST or not? We were like, uh, we had to debug it a little bit, but it was a little tricky, you know. It's like, okay, this is asking for it, someone make a mistake. I don't know how big of a deal it is, to be honest. Like, I used— sometimes it's a big deal, sometimes not.

I mean, oh, is there supposed to be a space like right after this last instruction?

I think so. I think so. We were going back and forth. It's like, we don't know. I mean, we're like, okay, I think Axolotl maybe adds a space there potentially, but I don't think he does.

I think he just grabs it, the raw thing. And that's actually interesting because I saw a PR from Charles that came through yesterday. Was it yesterday?

Where he was like revising some of the other, like doing some revisions.

And I noticed that like he removed a space from a lot of the from a lot of these YAML files at the end of the instruction. But now that, that like makes more sense now. Maybe he didn't realize that that was intentional.

Um, yeah, this is something you gotta be, I think, careful about. Yeah, so just like, it's like, um, you can't control the output, I guess. Like there's no way if you wanted to template the output in, um, it just will dump it at the end or concatenate at the end always.

Yeah, it just concatenates it at the end.

Yeah.

Gotcha.

There's actually a question here that I actually want to know the answer to myself because I get confused sometimes. And the question is, what is the difference between the chat template and datasets type parameters? Yeah. I just kind of like copy configs and kind of go with it, but I don't really— yeah, I didn't take the time to learn.

Yeah, so if you look at like, um, Llama, right? So if you were to take a look at the tokenizer config here, you'll see that it, um, do they, did they add a chat template here?

They didn't have one here.

Um, I believe if you go here, right?

So they have a chat template here in this. So what, when you set a chat template, what it does is it like, it will add this to the tokenizer.

That's really what it mostly is. It will add this to the tokenizer automatically.

But also in specifically some, like, so like, this is where we don't have coverage across like all of the configurations, right? For like, specifically like the SharedGPT types, right? That is like managing.

So this is where one of the things where I really want to rewrite, actually do want to rewrite a lot of these prompt templates and sort of think about it more as an input and an output, right? So you have a dataset in a specific format, and that is an input, and you need to, like, get that into a common dataset or a common sort of structure, like, call it, like, message list, right? Whether or not it's single-turn or multi-turn. And so I'm starting to think of, like, these datasets should really be, all right, we're taking all of these datasets, and we're doing a transform into some common structured within Axolotl, and that's, that's like, that's the type, right? So that's like SharedGPT where we're taking it, we're parsing it, we are understanding what it looks like and getting it to something. And then sort of the output really is ChatML, right?

So I believe the default for SharedGPT is like ChatML already, if I remember correctly. But if you were to set chat template outside of it and you don't explicitly set con—

I believe it's conversations on ShareGPT, it will go in and add it.

But if you change—

if chat template was like Llama 3, it will automatically go and set anything, um, any of the ShareGPT ones to, um, Llama 3.

So that like, um, so like it would—

if you use Llama 3 chat template out like higher up in the configuration, up above here, you normally see like I in a lot of the, like, copy-paste ones, you'll see chat template right above datasets, and then it will automatically include the— set the conversate or the output type as Llama 3 or Mistral Instructor, what have you, in the output as the output format.

Does that make sense?

I think I have to try it. I guess like I ignore all this stuff because I'm usually fine-tuning base models. And then what I just like, I'm like, I just use the same. I just use like one format all the time because I don't care. It's my model. It's my template at that point. And you know, I don't care about, there is no previous format that I have to worry about.

Yeah. I think, you know, I think a lot of people sort of like land on just ChatML. And I think just because it's relatively standard now, I know there was like a big fight when we sort of were pushing ChatML like on, on these models because I think a lot of the inference solutions didn't support it sort of like, I don't know, about 9 months ago, it was sort of like I think that wasn't well supported.

Cool. So we answered that one.

So Artem asked about no-ops for validation. I don't—

there aren't any no-ops. I will say that as far as unexpected side effects, I would say the biggest ones are if the base model is in the base model itself is in like, if you find that this isn't, I remember, I believe some models are saved in float32 that you get off Hugging Face or if you were to get like the raw weights for like Mistral or I believe like Llama 2, they were in like float32. And if you were to like do load it and then save it and you were using, let's say, float6, or bfloat16, it would save it in bfloat16. And yes, you would get sort of those like rounding errors. Also, if you were using— if you were on a machine that didn't support bfloat16s, right, and you had to use float16s, you would get sort of rounding errors there, you know, or not precision errors there when you go to save, right? So that is like one thing to keep in mind. But as far as like any existing NoOps that are to like do any validations, we don't really have any of those right now. I know that hasn't really been an ask up until now.

Being called on stage to ask a special question.

Yo. Hey. Yeah, well, could we go back to the config for the Mistral Memorize that we were looking at on GitHub?

This one?

Yeah, this one. Yeah, I actually wanted to double-click on the point about the space that Hamel brought up, because this is something— yeah, we went like, okay, so if you— so that format there used to have a space at the end of it. And, um, yeah, go look at this PR. You can see it. The thing is YAML will remove spaces at the end of non-quoted strings. So actually that space was not showing up in the things that were going into models. Like depending on your YAML parser, maybe that behavior would be different. So if you wanna have a space at the end, you actually need to quote it. And the reason why many PRs would like remove that space and then I went in back and added it is because our like YAML formatter, like your linter will remove that space 'cause it knows that in YAML, it's not supposed to be the case that there's a space at the end of any literal values that are typed as strings. So the, the, the like repeatable lesson from this is trust fucking no one, right? Look at the tokens, look at the inputs, and that's the only way to know what is like— there are so many places where the strings can be like mishandled, that you have to look at them as they're about to go into the model and then as they, and then as they come out. And that's the only way to be sure about this behavior.

Yeah.

Cool.

Yeah. I mean, that gives me some like ideas on sort of maybe starting to handle user-defined chat templates, right? In the YAML file.

You could do some things like that, but That's another topic for another day.

Let's see.

Oh, and then Charles has another question.

Very—

Yeah, by the way. Charles, you just want to ask it?

Yeah, yeah. Let me see if I can get it to show up again. Anyway, I'll just ask it off the top of my head. So yeah, so now, a little bit unclear how to ensure that the tokenization is handled exactly the same at inference time as it is at fine-tuning time. Like, I, for like, just setting a template doesn't seem like it's enough because there's also the like masking that Axolotl does. So I'm wondering if there's like some format or some workflow that I can use to ensure that the same behavior is triggered when I'm using VLLM to do inference and when I'm using Axolotl to do fine-tuning, for example.

I mean, so masking shouldn't affect inference because masking is just for the loss calculations, right?

But don't you tokenize them separately? You tokenize them, then mask their values. But if you were to take a raw string that had the same format instructions, and then tokenize it like as a concatenated string as opposed to tokenizing each string separately and concatenating the text.

Oh, right.

Yeah. So, so there are, yeah, there are some, and Hamel Husain has a blog post on that, right? And that's why I do recommend like thinking about using the chat template strategy.

So sort of to address that, we did add like a chat template prompt strategy here where you can like specifically use a chat template.

Okay.

I don't know that it's well documented, but if you use like this chat template, it will, it actually handles everything almost near perfectly because it uses like the ad generation prompt. It generates the entire string and then like goes back and I think this is much closer to like something that shouldn't have errors because like Um, you have the prompt IDs and the input IDs, and then it sort of like backs it out.

So then do I then need to— yeah, do I need to also bring this same like prompt strategy into my inference engine? And, or is there a way to guarantee compatibility with the temp, like the prompt construction technique in my, in my inference engine. Like, if I use the same— if I use chat templates because they have these like specialized end-of-sequence token IDs that are handled special-cased, then I don't have to worry about tokenization collisions between like the inputs and the outputs, right?

So like, if you're using the chat templates, you, you shouldn't have to, because as long as the inference engine is using the chat template that's defined in sort of like the tokenizer config, because what happens is if you set the chat— if you set chat template in the YAML file, it will, it will add in that specific chat template that you chose into this tokenizer config for you.

So you can change, like, if there happens to be one set, you can change it, will override it to whatever the new chat template is for your specific fine-tune.

And then this is, this is maybe a noob question, but it's like these tokenizer configs are also what are consumed by many of the, like, mainstream inference engines because they use the same, like, Hugging Face tokenizers, right? Or they at least use the same tokenizer config format.

Run that by me again.

Sorry, a lot of times, yeah, I don't trust any of this shit. Like, I'm just like, look, like, I'm going to do the prompt template myself. Like, I don't trust anybody because, like, when— as soon as somebody else is, like, constructing your prompt I've always found something going wrong in that.

Yeah.

That's fair. I think maybe like some kind of test that checks and flags differences in tokenization between your fine-tuning and your production is like probably a good way to like operationalize you, because you could make a mistake yourself. Like, you know, Hugging Face aren't the only people who write bugs in the world, you know? Um, like I've written plenty of bugs myself. So, but yeah, the question I was asking, Wing, is these tokenizer configs are what you use, and then they are also what the inference engine uses.

Yeah. Ideally. Yeah, ideally it's what the inference engine uses. Assuming, I don't like, I don't know to what degree like VLLM like will check the chat template these days.

I don't.

I honestly don't know. Like, I think that, and on, and probably because it's just VLLM, it's actually, that is not an application layer thing, that is just an inference thing. So it could even be that, um, well, I guess if you have to look like the VLLM OpenAI endpoint, right?

Um, oh yeah, I mean, it's not perfect, right? Like, even when you use Axolotl, because of that, because they're doing concatenation in the token space and not in the string space, even when you're writing your own template, like, even if you do it perfectly, if you're concatenating in string space, which you often are, it might not result in exactly the same tokens. You just have to decide whether you care.

Yeah, it's like if you're off by a space or it's, yeah, it can, it can, usually it's not too bad. It's usually just that space.

And the way I sort of like look at it is, Because like, I think we— I remember when you were talking about it, and it seemed generally that even though the tokenization wasn't perfect, the model still understood how to like generally work.

So it wasn't terrible. I'm sure there could be some accuracy improvements if the tokenization were perfect. So yeah, it's definitely— You should check the work and just, you know, and if something doesn't seem right, send a PR my way.

Got it.

Um, a lot of discussion on this topic in the Axolotl community, like, about tokens. I actually was, like, really paranoid at first. I was like, oh my God, like, it's gonna go wrong. But then I was like, I was trying to prove to myself that it, like, these small perturbations I'm still paranoid about it, to be honest, but like, in practice, like, it didn't affect me for some reason. Like, so I was like, okay, not to say that that's a good thing per se, but I'm just— so then I just like these small artifacts, I just kind of like decided whatever, I'm trying my best. But you should be paranoid for sure.

Yeah, it'd be interesting to see if the model during training learns something that would allow it to interpret, um, the content the same whether it's concatenated in string space or token space? Because like, it could learn that— it could conceivably learn that like an S and then an S token, then an S token means the same thing as like an SS token.

Well, see, I don't know, this is like fine-tuning, right? So like in fine-tuning, your model is always seeing one thing.

Yeah, that's actually like—

Surprising that it works when it's always seeing a little bit of a different thing in inference.

But yeah, we don't know, we don't know what was learned at the base level, at the base model level. And then like, that's going to have a big influence on how your fine-tuned model works.

Yeah, my, my actual instinct as an ML engineer is actually to introduce tokenization errors during fine-tuning as a form of like data augmentation. As, and as a form of like additional robustness. So it's like, it's, yeah, this is like roughly what it would tokenize to. Like, it's plausible to me that in the downstream inference system, this kind of tokenization error would occur. It'd be challenging to get that to work precisely with the masking stuff, but, um, yeah, it's almost like regularization to some degree. Yeah, yeah. It's also, I mean, like with images, you know, when you do fine-tuning or training of image models, you put in all these images that you would never see in like in inference time, just because you expect that like various other kinds of bad things can happen to your images. And that's— otherwise you end up with a brittle pipeline.

You're basically making your model more robust to, to a common user error, for sure. That makes sense.

Um, let's see, uh, there's a question about multimodal. Um, we started like a long time ago when LLaVA first came out to try and add at least sort of the um, uh, the, the first step of LLaVA, uh, to create the model projector. Um, that never got merged, and I think LLaVA support got added to Transformers, but it's more of a matter of like doing the dataset handling at this point, right, for images, and like pipelining that into Axolotl.

So we never— so it's like on my list, but it is not within sort of the, the time capacity that I have right now to get it working. Um, I, I'm hoping to get it working soon, but you know, soon means like in the next 3 months for me. Um, uh, so it also— it's high on my list, but so if—

but if anybody wants to write a PR, more than welcome.

Yeah, I feel like the, the dataset side is the part that's going to be a little bit more durable, right? Like, because we're already starting to see a move towards native multimodality where the different modes are represented as tokens, and then you route the input and output tokens appropriately. Um, and that's like quite different, I think, from the way that like multimodality is handled with something like LLaVA. Uh, but so that might— so like the details of how you handle things, the tokenization and model level might change, but the datasets, they're the formats. That's something that sticks around for longer and is reusable across models. So like, that's the place where open source development is most valuable, in those things that are, that are like kind of stickier. So yeah, that's where I'd hope for people to make contributions.

Yeah, that would be great.

Yeah.

Um, also the training scripts in the LLaVA repository are easy enough to use that, um, there's probably less of a demand than in some other place. Like, fine-tuning LLaVA's not bad. Yeah, even today.

Yeah, I think, yeah, like this was for the pre-training step, right? And I believe it was like, yeah, it was definitely a little bit janky with like handling the data, pulling the data in.

I don't remember, I don't remember where it was.

Build data collator. Yeah, but anyways, It, you know, I'm probably, this probably just going to close that one out and need to start from scratch anyways for that.

Cool.

Is RLHF still a common fine-tuning technique?

I think yes, generally, and I think you're going to start to see it be more and more like be the mainstream as sort of the limitations of like accuracy and quality that you can get from like supervised fine-tuning starts to hit some sort of like wall, right? And if you think about it at the end of the day, like, if you, like, if you look at OpenAI and Anthropic and all of those, right, the, their models are very good. Yes, data to some degree, but also data quality. And a lot of that data quality comes from RLHF and those sorts of techniques.

So, so I, there are probably two ways to interpret that question. So one is like reference tuning, which would include DPO and, and whatever else. And the other is like literally RLHF. And do you have the same answer for both? Is the answer you just gave the same for both? Or are you, like, one candidate answer would be like, we think there'll be a bunch of preference tuning, but it won't be RLHF.

Yeah.

Are you just trying to like say the difference between like RLHF and RL-AIF and that sort of like difference? Or just very explicitly like, like taking human preferences from like the outputs.

Yeah, I guess I don't know how they meant it, but I'm curious whether— so there are some techniques, there's no RL in them at all. So DPO being an example. And so when you say you think there's going to be a lot more RLHF, one way of interpreting that is I think there's going to be a lot more tuning that actually involves RL. And another is, I think there's going to be another possible interpretation of, of what you said is, I think there's gonna be a lot more preference tuning, but some of it might be DPO. I'm curious, like, uh, do this— do you think there's going to be a lot more RL-based, uh, preference optimization, or just there's gonna be a lot more?

I think it's gonna be a bit of both. I think they sort of like come hand in hand, um, and it's just gonna be a matter of like what resources you have available.

So, yeah.

Cool.

And I had a question also on that front. Um, with DPO, aren't you limited to like basically one turn kind of fundamentally? Whereas with RL, like with like actual RL approaches, you have, you can have multiple turns in an episode and you can have intermediate rewards throughout, whereas with DPO, you just have to like maximize probability of being selected, right?

Yeah, it's maximizing for each one. And yet it's like, we don't have like multi-turn support, but you can construct a dataset that supports multi-turn, right? Um, you just have to, um, you just have to like be a little bit fancy about it. We could like abstract some of that away as well.

Um, yeah, but that's— it's just like a much more robust way to train a model, like training on policy with RL methods as opposed to just training with like a single reward at the end of the episode. It's going to be more data intensive because you're going to have to do your temporal difference learning to learn anything about the beginnings of conversations if the signal only comes at the end. And I think there's some research results that suggest you can get much higher quality alignment with the deeper RL methods instead of just DPO.

Yeah, that makes sense. Yeah. And yeah, I think there were several, I think a couple papers in the last couple weeks that shows like sort of the PPO is gonna be much better, but I think it's like, it's definitely much easier to not have to like build that reward model, right? So I think there's definitely a move to sort of go that route. Yeah, and it's also like, I guess depending on the size of the reward model, right? If you can have a relatively small reward model, there's, sort of having to deal with having that reward model in memory too, right? So that can be very size dependent and sort of, um, independent on like your availability of compute resources.

Um, someone's asking sample files for PaliGemma and Phi-3. I believe the Phi-2 works with Phi-3.

You just swap the baseline. I don't know that there's anything specific on Phi-3, that would not work.

Let's see.

I think maybe with Phi-3, you might have to have trust remote code, it looks like.

I don't know if they've merged. Are they waiting for a merge into Hugging Face or anything? It's been integrated until the official version is released. Yeah, so maybe you might have to have like trust_remote_code is true in the YAML file to use Phi 3.

As far as Polygema, I haven't had time to use Polygema, so I couldn't. But I'm guessing Polygema doesn't have remote— doesn't need remote Trust remote code since there's no nothing here for that.

So I would just give that a try.

I—

this is their— is this their, their like recursive model, or is this a, or is this a multimodal model?

Vision language model.

Yeah, I would say you could probably do like the LLM side, but until like similarly like with, you know, until we have multimodal dataset handling going, Polygema might be a little bit off of getting support for We got a couple questions with 7 or 8 votes.

Should we try and take some of the most highly voted ones?

This isn't sorted.

Sorry.

I was sorting by most recent.

Sorry.

All right. Got it. Dumb question.

My understanding is conversational datasets are most effective for fine-tuning.

Any tips on setting these up, or can we get enough performance with just QA pairs?

So like, can we like talk about that? Because I don't know if that assumption is true. Like, our conversational— I don't know if conversational datasets are more effective fine-tuning or not, to be honest. You know, I don't necessarily think that they are.

Yeah, my opinion would be like, you need to be fine— your fine-tuning process needs to resemble as closely as possible your production process. Like, that's not just the like fiddly details about tokenization. It's overall like during training and fine-tuning, you are demonstrating this data generation process to your model. And if you want it to be good in production, they should, like those two things should be as close as possible to each other. So if you're like, if almost all, like if even if you have a support bot, if almost all of your user queries are like one turn, then you only need QA pairs. But like at another level, it's like if you have written something that is just like call the language model, get the result, send it back to the user, and then every interaction starts over from scratch, then you definitely don't, you don't need a conversation dataset. You don't need a conversation format. There's lots of rich information in those conversational datasets, but they're also trying to train the model on a different thing. Thing than what you're doing for like a RAG QA search system.

Yeah, and I think I know where the confusion is coming from, uh, from Sanyam. Um, let me just share, uh, oh, I can't.

Okay, yeah, let me stop sharing.

Just to get more intuition about conversational datasets, I really recommend that you instruction tune a model so you kind of have some intuition on what conversation dataset means. Like, it'll give you some grounding and be like, oh, like, that's just a— it's just the way the data— the model was trained. It's not anything special. Nothing special about conversational datasets versus another kind of dataset. It's just the way that you train a model. And like, if you do this kind of like instruction tuning, it'll give you a good— it's a good foundation on which to understand like fine-tuning. So like, I would recommend I would recommend doing this if you don't really quite understand, like, what is conversation, like, where is it coming from. It'll just, like, set everything straight.

Um, there was one question I wanted to—

all right, um, let's see, there was one question that I definitely wanted to Answer.

And—

um, quick interjection while Wing's looking. Um, I would like second Hamel's statement, and I think really pre-training is the part that's like best understood, easiest to understand, and where a lot of these issues end up coming up anyway. And the nice thing about pre-training is there's a very clear metric for success with, um, perplexity and compression. And so you can take a base pre-trained model and then like fine-tune it on just completions of a different dataset. And you should observe increased ability to compress that data. And that's like, that's like a sort of like Hello World or test way to do your like fine-tuning process. And like, unlike conversational or instruction following datasets, there's like very clear, there's like very clear robust metrics to connect to. So maybe the next edition of this course will have that as an exercise.

Also, is there a way to export all of these questions?

And maybe— I don't know that we'll get to all of them, obviously, but I can try and like take a pass at getting some of these answered later on. Do you know if there's a way to export these questions?

That's a great question.

Someone's always screenshotting.

Someone had a link to a Zoom help article in Discord that said you could. Now I have to find that in the haystack, but I think there is. I've seen someone.

It'd be a real miss on Zoom's part if you can't, so I would hope that you could. While we're talking, I'm gonna So I'll start on the top again.

Some concrete examples for training datasets for fine-tuning of completion models. So I mean, your completion models are going to be just any dataset.

You could train story generation, maybe if you were trying to write stories and you just wanted to keep— I think so. I mean, any of the pre-training datasets are usually going to be very similar to the datasets you would use for completion models. Typically, all of those datasets usually have like a text field and it just like trains on everything in that field.

Instruction models, those are like your QA pairs.

Most, a lot of, even a lot of what are like deemed some conversational datasets, they may just have like single turns. So if you can filter all the data at single turns, it's pretty close to an instruction model. All the alpaca datasets are going to be instruction models.

Then multi-turn chat models, I think OpenHARMONY is like a good dataset for chat.

I believe a lot of those do have multi-turn, and then sort of like, I think also the CappyBar datasets might have multi-turn as well. Yeah.

Let's see.

If I want to fine-tune Llama 3 that works smoothly with a library like LlamaIndex, LlamaIndex, okay, which is tightly integrated with OpenAI template. Which prompt template name should be used next? Llama in this case. So I believe even if you're using it with LlamaIndex, it doesn't necessarily matter which template you're using because the abstract, the OpenAI abstraction is sort of the messages abstraction, right? So Ultimately, the inference that you're using underneath it with LlamaIndex is going whether or not it knows how to use whatever chat template you've defined to sort of convert those individual messages into your template. So I don't know that there's necessarily one to choose. I would personally just go with ChatML. I'm sure Llama 3, like the Llama 3 in them, should work. I would stay away from the instruct ones because they don't seem to be very well defined for multi-turn. That would just be my takeaway on that.

Future directions of Axolotl.

What areas do you need contributors' help in? We need contributor help on a lot of things. Like, I think a lot of it is just like the issues just keep growing and growing.

I personally can't keep up with them.

So if people like as they learn and, you know, find things that they can help answer other people's questions on, that's always helpful. There are definitely sort of like if you looking for things that you want to like, if you think to building features, you know, there's the axolotl- dev channel in our Discord. Um, if you pop in there, say you want to like do something, I can, I can probably definitely find something for you to work on.

Um, yeah, so just to, just to show you a little bit more, do you talk about a little bit about what you're building like commercially even that may be related? Do you still do support contracts for companies? I know that in the past you, you have some. Now do you still do that?

Yeah, yeah. So I still do, you know, we're—

if you have like a need where you need a smart contract, professional services, those sorts of things, you know, reach out, either DM me directly on Discord and/or email me at— I think it's— so you can email me at like wing@axolotl.ai. And then sort of, as you know, we're working on sort of more turnkey platform for users. So you can do like— so you don't have to go and track down GPUs. Very similar to Modal to some degree. A lot of that will be built around supporting also like DPO and other eventually like PPO and dataset pipelines. There's a lot of things that we're trying to build into sort of an Axolotl platform where you can just do things like install Axolotl CLI, do Axolotl cloud submit in the YAML file, and it'll just like go and handle everything for you, handle all your secrets, like assuming you've set your secrets once. So very similar to like some of the other cloud platforms, but more tightly integrated with sort of Axolotl OSS and should have tighter, tighter loops on getting those deployments out.

Let's see.

Well, you know, if, if you need somebody to provide you a secure container runtime with GPU acceleration and snappy cold boots underneath that software platform you're building, I think you know who to talk to.

Yeah, yeah, for sure.

My, my thing was always the, um, is the spot instances, right?

And so that's like, if there were a way to, you know, especially for longer ones, and not have to like worry about sort of getting preempted, right?

That was—

but you know, yeah, well, we can talk offline about that.

Yeah, GPU checkpointing coming soon. Which really makes it faster when you get preempted.

BRAM estimation, given an axelmodel config, that is like, if someone wants to write a tool for that, that would be great. I think that, yes, we could like, there are some basic estimates like you can get from using any of the existing.

I think Zach talked about that on Tuesday as well. Sort of built into that. There's a lot of like papers on sort of LLM math and, and what that looks like. It starts to get a little fuzzy around the edges when you start to do things like FSDP and DeepSpeed and those sorts of things. So that's— and I think at the end of the day, most people are using those anyway. So it's like it just makes it somewhat less useful, but usually it's more around the I will say that if you're using like FSDP and, and, and or DeepSpeed, like increasing your batch sizes doesn't affect your like, you know, your VRAM use as much as if you weren't. So like, I think you definitely get like huge stepwise changes when increasing like your micro batch sizes from like 1 to 2 to 3. And the, those steps are, are not as large when using FSDP or DSP from what I've noticed. And Charles shaking his head, so I believe he agrees with that.

Runs, infancy vibes. So while training, can you pause? I don't, let's, so pausing, so you could like, In theory, like if it saves a checkpoint, you could like, you know, break out of the training inference against the checkpoint and then just restart using the checkpoint. That's one way to do that for if you want inference against the Vibes. If you had sort of, I have like, I think I talked about this on Tuesday also about like if you had a specific training data, like eval dataset, you could specify and it would like—

there's like an eval table that it can run predictions against, and just, it'll spit out the— it'll just spit out the predictions for a given prompt and then save those to Weights and Biases. You can do that as well. The downside to that is it doesn't compose well with sort of like the existing like running the existing eval loss.

Could you do what you said about doing it while training's happening by just starting a new process, potentially in a new terminal?

Yeah, yeah, I guess you would have to—

your concern would be whether or not there you have enough VRAM on that same machine to like reload it up and, you know, which is probably unlikely if you're already like pushing near the max anyways, because you're trying to optimize, you're optimizing for your training anyways. I would say that the other option could be like some sort of callback.

So I've actually thought about doing this is like you could add a callback that reads like some queue or file on your thing.

And then like after on each of the eval steps or something, it runs predictions against what's in that file or like run— so like you can update that file live and get like real-time predictions on the sorts of things.

See, Hamel Husain, um, let's see, um, so that was answered.

Best practices for friendly and varying prompt templates, like if you can see what it look like in the code to inputs and outputs. So here I can do a really quick demo on this one, um, so I— and I think this got— was talked about also on, um See, this was talked about also, but just so people can see it again.

Share. That's this thing.

So like if we were to—

oops. ShareGPT.yaml. All right.

So if you just do like python -m axolotl.cli preprocess sharegpt, and then just add a— toss a debug at the end. Don't toss it in the middle, otherwise it won't pick it up properly.

It should like preprocess this, preprocess the data.

Yeah, we definitely talked about this one. That is too big of a dataset.

1, 2, 3, 4, 5, 6. Yes, ta-da! Messages, dressing.

This may not work.

We'll just— all right, Python. Examples. 3. We'll just do this one.

So it'll, it should like, it'll tokenize all of the, all of your data and it should just spit out it.

I believe it takes a random row.

Oops.

And we'll show you the, anything in red is like not counted towards the loss.

Anything in green is what you expect the model to output. So for example, here you can see like the beginning of text, the tokens below is it, the system prompt is here, the instruction is here, sort of like the response, and then what you expect the model to output is in green, as well as sort of this, you know, this sort of end of text, which just indicates as the stop token. So Yeah, that's like an easy way. And I think that was talked about a lot already.

I'm going to need to drop in a minute or two, which I think will end this, but I did find a way to export the questions so that we can put them in, we can put some of the unanswered questions in Discord and Yeah.

Yeah. Are you, are you, so you, you were able to get the, extract the questions out already?

Well, I found a way to do the, do it after this closes.

Oh, okay.

So Zoom produces a report at the end of all the answered and unanswered questions. So I'll extract those and share them with everyone.

Okay, great.

Yeah.

Comments on Axolotl versus Unsloth. So I would say that Unsloth is great.

I, I'm a fan of Unsloth.

So sort of the things to take away from Unsloth is it's very specific.

A lot of the optimization, the free optimizations are going to be LoRA only, and I believe it's limited to like 2 or 4 GPUs that you can use, which is fine for most people in most use cases. We did recently integrate like their open source kernels for LoRAs. For Llama only. I haven't had a chance to go and like do all the patching for Mistral and the other, like whatever other architect, Gemma, and I think one other one, and Phi, I believe that they support. So yes, you can get some of those memory gains. You know, I think generally, we, there are enough speed improvements in Axolotl, like with sample packing and those sorts of things that You know, it's really, it's, you know, pick the tool that works for you. At the end of the day, like, if using Unslot is like, you're still, you're then on the hook for managing a lot of the prompt, you know, the prompts and everything else that Sort of Axolotl takes care of for you. So it's like, yes, there are definitely like ways we could try and integrate the two and get the best of both worlds, but it's just a matter of like tool choice right now. And we have to pick, sort of the things that we're good—

Axolotl needs to be good at.

And for right now, it's like we're eking out the performance that I know how to eke out. And then also, um, I— for me, it's like, as an application developer, right, these are the, the things of like managing prompts and all of those things are more important for me. Like, those are the, those are the important use cases, and performance are, are things that we can work out later.

And Yeah. What is the best way to train a quick and dirty fine-tune and verify your model works at inference time as expected? I would just like do like Tiny Llama, like take a Tiny Llama example.

Those work pretty quickly. And then using like the Gradio inference that was talked about on Tuesday, and it should be in the documentation.

Function calling fine-tuning, specify custom format.

So I know that like the NUS guys, they did, they used, they have some function calling datasets like the GLaive datasets and stuff, and they use the SharedGPT. Definitely let me dig up, if I remember correctly, like there's some like role outputs, you have to specify like a load a specific version of the SharedGPT for the roles to handle the function calling roles and those sorts of things. But I can dig that up and just make— if someone tags me in Discord, I can try and remember to answer, find that one.

Let me see if I can find like fine-tuned vision models.

So I—

we already talked about this.

Right now there isn't support for like vision and multimodal models, mostly because we don't have dataset handling yet for images.

And let's find one last one. Okay, Chris was asking about like the preprocessing and seeing what it looks like when it's packed in batches. So that one's a little harder. We don't do that.

You could probably manage that by modifying the code and dumping out the actual tokens in the forward call of— if you maybe modified the Transformers code in the Llama for the model, Llama for CausalLM class in the forward pass, you could print out the input IDs and probably log it there.

It's a little difficult because a lot of sample packing actually happens at runtime and doesn't happen during the preprocessing step.

And the reason we do that is when we use— we've used sort of a random sampler and a seeded random sampler for packing. So it will pack differently on the first epoch versus the second epoch. Those sorts of things so that you don't have— so that the same, like, samples don't get packed together from the first epoch versus the second one versus the third one.

So that's like the main reason we don't— we don't—

aren't able to sort of debug those sort of at preprocessing time. All right, cool.

So That's all I have time for right now. Dan will export these questions.

I will work, you know, I will do my best to answer the rest of them as best I can later on. Thank you everybody for showing up for this and taking the course and your interest in fine-tuning and Axolotl and X-Wing. Yeah.

Okay, great.

Thanks, everybody.

See ya.