FSDP, DeepSpeed and Accelerate

What are your feelings on Axolotl versus HF Auto Train?

There's a wink emoji. Who is this? Because it's either my coworker or you know who from Axolotl. They solve two different problems in a way, right? AutoTrain is agnostic in the sense of like train whatever you want with the data you have. And it's also a slightly different API with what it looks for. And Axolotl is very high level, train a model for basically text right now, as Wing was talking about earlier. Do it quickly, do it fast, following a pretty loose guideline towards what you can fit. So there are two different prob— there are two different solutions to a problem that can have overlap, if that makes sense.

I've never used HF Auto Train, so I was actually— I prioritized this question because I was hoping to learn, and I think I did, uh, what HF Auto— Hugging Face Auto Train actually does through your answer. Um, and I personally did that. It almost sounds like, is Auto Train really doing more of like the smart search of parameters part of the problem?

Uh, oh boy. Oh, the Auto Train talk, uh, soon we'll be able to talk about that more because I don't know. Um, my extent of Auto Train is helping debug Auto Train when Accelerate breaks. So that would be a good question to save for the AutoTrain talk.

Yep.

Is my official answer.

Great. I'll count that one as answered, and I'll let you start picking some.

Sure. Since many of us are new to LLM fine-tuning, how do you think the typical learning journey looks like until one becomes an effective LLM engineer? Here. Just tweak with shit. Like, just play with code, build models, do things. Because you can spend ages reading and reading and reading and reading and never touch code. And someone who's just in their backyard doing these things for fun on the side will have more practical experience for you. That sort of was the situation 3 or 4 years ago when I was learning fast.ai. And I still believe that situation Train models, get comfortable with the code. The depth of expertise will come with respect to the amount of time you do it. Even if it's just running the same, like, Axolotl command and just looking at the Weights and Biases graph for different datasets, you will learn something there that I probably won't because I haven't done that enough yet. So just start playing with things. And sort of the same advice that we started giving with fast.ai back in the day, make it public. Make your learnings public. Your model's available. Your teaching's available. And be open to criticism from the people that know a lot more than you to learn from it. And be very humble in that regard is, I guess, my answer to that.

I wanna push back for a moment just to get, like, I, maybe there's something interesting in this. And that is when you are just fiddling, which I, I quite like doing, you learn by seeing what works or what happens and what doesn't happen. And if we were doing, there are many domains and conventional machine learning is probably a great example where you try something, your score goes up or it goes down, and you're like, oh, this improved it or didn't improve it. And with generative AI use cases, and I think language in particular, sometimes you can see what happened to your perplexity score, but it's a little hard to know, like, do I like what just happened? Or like, what did just happen when I, I don't know, changed my learning rate or changed whatever? Um, do you have any thoughts about like how one gets feedback as they're experimenting in order to make sure they're learning?

Yeah, talk to the people that are doing the cool things. Just send them a DM and say, hey, I'm doing this, I don't understand, you know, what this particular graph means, or post it on Twitter. Like, that's— if you go through my last Twitter feed from the last month, I'm pretty sure I made a tweet that was quite literally just, hey guys, here's my loss graph, does this seem normal? And because if you're only judging yourself and going off of yourself and your own knowledge you're going to think everything's fine because everything looks normal, right? And so that's where communities really matter, especially like this community that is now forming, because we're all here to go learn this thing, right? So we can all sit there and go, hey, here's my graph, what are your thoughts?

And I think it's also important to like, um, pick a small project. It's really good if you have LLM production already and you have some traces you have some logs about what, how your LLM is responding and what the user is asking and all whatever. We'll go through that in the next lecture a little bit of like how to collect those logs. But if you can take those and just fine-tune a model and like start with like a smaller model, start with LoRA, it's actually kind of hard to mess it up. It was like really nice, like LoRA in a pre-trained model, use Axolotl, you don't really have— it's not like a— it's not you have to get the hyperparameters exactly right necessarily. Like, it's kind of— it has a— yeah, it's like kind of this random forest feeling where you just like point the model at the data and it kind of works. And if you get that positive reinforcement, it really helps if you get it fast.

Yes, it's, it's easy as I look at my graphs from the last week.

Going back.

It's also easy. Yeah, you're good.

You're good.

I was saying, going back to our talk from when we were with Wing, parts of it might be easy. The part that's hard is actually dealing with tokenization and prompts and making sure that you do the same thing at inference time as training time. And that is like the opposite of, that's like very hard.

I think what I spent the last hour making sure that I could get my prompts right for inference after doing my Axolotl training. So it's a learning curve, and some of it's easier than others, but that's part of the game.

There was a question in Discord, which was— and I want to say one thing before, then I'll hand it to you guys for an answer— which was Uh, something like, where do all the cool people hang out that I can use to get feedback? Um, I think that despite the immense amount of hype, the community of people who are actively like doing stuff and sharing it, even like community of doing it, people who are doing interesting things, is smaller than you would expect and is sort of in its infancy. And as a result The ability to get feedback is much, much greater now than it will be a year from now when there's more people doing it and then it's harder to get people's attention. And, um, so we could talk about like Twitter and this, I think our Discord, a bunch of places that you can think of, but I, I just want to say like, I think it is a huge, huge opportunity for us to, if I can sound like a salesman for a moment, like to get in on the ground floor, by which I mean like people are not— people who do cool stuff like Zach are not so inundated that they totally refuse messages from anyone or anything like that.

No, this community, like, it's been a very humbling experience because I sort of wasn't doing community stuff for a bit, that it's— we're all people, you know, and a lot of us don't have egos. And especially right now, while it's a fine group of people like you were saying, make use of that, you know. Talk to the peoples that are coming out with these models and evals and ask them questions in Twitter DM on their responses. More likely they'll not— they'll answer whatever your question is as long as you like phrase it right and don't, you know, give them not enough information. That's a—

but you definitely should put in the work. So like, don't— yes, just like, don't just DM someone like whatever, like, you know, like tell them what you tried, what didn't work, what you researched, but then you got stuck. Like, show the person that you are trying. So from time to time, I'll get a DM from someone that they just didn't try to, like, read the documentation, read their error message, read anything. And then in that case, many times I have to ignore those. Um, so don't, don't be that person.

Uh, I'd be actually super interested to hear Zach's answer to one of the most upvoted questions, which are like, what are some good public datasets or Kaggle challenges for training or fine-tuning, uh, LLMs? Like the Titanic dataset or Iris or Penguins or whatever, but for LLMs.

That's a good question that I'm not 100% sure on the answer for, and I'll sort of explain a little why. So my Llama 3 fine-tune is based off of the StarCoder 2 self-instruct paper. And essentially what that is, is using an LLM, using the base LLM to generate your completion dataset for you off of existing code on GitHub on a dataset called the stack. So like, personally, if I were to choose a starting dataset, that one's not terrible because you're generating it yourself or you have them available with known benchmarks that have end-to-end full transparent pipelines. So you know what's good, you know what's bad, and you know roughly where you should end. That's at least one that I know of that I'm currently doing to learn, but I'd be very interested to know what you guys sort of also use as a gateway sort of dataset and problem.

I like the instruction tuning one a lot. The Phil Smith. Am I— I don't know if I'm pronouncing his last name correct. Whatever. Like Phil Schmidt. He has like a fine-tuning Llama blog post. I think you say— I think instruction tuning or sorry, instruction tuning Llama. I think instruction tuning is a really good gateway because it kind of like gets your mind ready for fine-tuning.

100%. There's a— There's a ton you can learn in that. Let's see, what was the other?

I need to put together demos for potential clients somewhat frequently. And if you just go to Hugging Face and do a dataset search, like if I wanted a dataset about discussion of chemistry topics, I can probably find one, and, um, then like it's probably in a standard format, and then I just drop that link in the Axolotl config. Um, I would say that, uh, I mean, the same way that Kaggle used to be and maybe still is for tabular datasets, like the breadth of what's in, um, Hugging Face datasets is pretty great.

Yeah, one thing I would say is that synthetic Synthetic data is really nice for, like, one, there's a practical use, which is taking a model that's too expensive to run and getting it into a smaller model. So, and then two, you have a very clear data generating process that you're trying to mimic and you can run it, right? If you have a dataset you got from the internet, you can't like generate new data from that on the fly ad hoc. And that's really important for like testing whether your training is working or not. Um, so, uh, yeah, and you— so you can always compare it to this executing— this like model you can also execute. So that's maybe you run inference in a 70B model and you're fine-tuning an 8B or a 3B model. Um, and you just— and so, and you have a really solid gold standard because all you're trying to do is mimic that thing. Um, and then the second—

you're good.

No, okay, yeah, the second piece of info— the second advice is that you also want something that changes over time, like the dataset grows over time, because when you, when it comes time to actually deploy machine learning in production, you're gonna have like a stream of data, not like a batch dataset that you just run on. And so if you want something, so you like, just for example, you might take tweets and have GPT-4 classify them, and that's your dataset. And then a week later there will be more tweets that you can run through GPT-4 and classify. And now you can find out whether you have like train-test leakage issues. You can find out if you're like overfitting to the, to the like this week's content of tweets and next week it's something different. And that, so it's like kind of similar to Numerai or some of the like ongoing challenges where you need to like actually learn, you know, of like a, how to model a stream of data. And that also is going to really help you know what you need to do when it's time to build a production ML application.

Let's see. One of the questions I was interested in, in my experience, it's always been a pain either to use Accelerate or Torch. Can you clarify the best practices when using it? And is it worth using it when running DDP, FSDP, or DeepSpeed? I just remember when torch.compile launched, it was all hype, but then there was no example in xLoL that uses compile. And does it have any benefit for VRAM? Yes. I would not be able to do my Llama 3 billion fine-tune on two 4090s if I did not have FSDP available to me. Because essentially what FSDP allows is your two 4090s are not two 24-gig cards. They are, in a light sense, 1 48-gig card. And so it's critical that you do that, which go back through the talk. I'm pretty sure I talked very explicitly about the difference in memory usage between the 3 of those. Let's see.

I guess the question was also kind of about torch.compile, if I understand it right. It was like using Accelerate with Compile, using Axolotl with Compile. And I guess my feeling is that Compile is like a little bit more of an inference time optimization than a training time optimization, where it's like you have a very fixed graph. You, um, like the, maybe your, at inference time, often your batch sizes are much smaller, um, because you're only getting like one request at a time. And so the overhead from running a dynamic graph generation is too high, and so you'd rather just— so you want to switch to a static compiled graph. Maybe, that's my impression from how I've used the tools, but—

That's certainly how it used to be. That is not where PyTorch is going. PyTorch wants torch.compile everywhere, which mixed Right, because it's compile, it can have issues. For instance, they have a library called Pippi out right now that is native Torch pipeline parallelism for training that relies heavily on Torch Compile. So that's where things get weird and sort of experimental. And I expect that landscape to change dramatically over the next, like, 3 to 6 months. So for right now, I believe you can train on compile. How well it works, a lot depends on sort of how the model is laid out, I found, with just where people's issues have come.

And the— so the benefit at train time is that you're getting like sort of dynamic operator fusion?

Yes. So it speeds up train time. Mostly it's throughput optimizations more than memory optimizations.

Yeah, like, like operate, like kernel fusion.

Yes, yes.

Got it.

Here's sort of an interesting one. I have a question about training inference precision. I'm training on 3090s at home or thinking of renting an A100 on Azure. However, my inference setting T4s do not support BF16, but my trained model, trained based on DeepSpeak, uses BF16, seems to perform poorly in FP16. Any recommendation on here on how I can increase throughput? I would use vanilla Hugging Face inference, which has fake BF16, but the throughput is painfully slow. You're gonna get painfully slow. You know, it's, it's, they're optimized for BF16 for a very good reason. And that's especially a scary place to be in because we had a lot of issues with models that were trained fully in BF16, which broke entirely when they were upcast to FP32 again. That's why Trainer uses an autocast form to train everything in BF16. So that way it can be upscaled back easily. So, but normal FP16 is— and Sam might come in and smack me for this, but in my experience, usually a little less optimized than BF16 will be, so it will be slower. Let's see what else we have. Ah, I like this. I like anti-questions. What are the main downsides with FSDP? When is it really the not best tool for your job in training and fine-tuning, such as hardware configurations, LLMs, diffusions, and vice versa? Basically, FSDP really shines when you have big models, when you're training big models, or if your model barely fits on one GPU and you have two available. Because now, essentially, what you can kind of do and imagine is the model is fully on one and the gradients and everything else are just in the other GPU. And so you have a ton of extra VRAM that you can go use. Now, FSDP versus DeepSpeed. DeepSpeed is very configurable. It has a little more freedom than FSDP would when it comes to like offloading certain things and having certain device configurations. FSDP takes the all-or-nothing approach. So like, I'm pretty sure DeepSpeed, you can specify like, just these layers should be offloaded to, you know, the CPU. So that way we barely don't hit out of memory. With FSDP, it's all gonna be done. So it's little situations like that where, does it all fit in memory? Use FSDP. If you have to do a little bit of offloading, might look at DeepSpeed, be that 0.3 or 0.2, and see sort of what optimizations could go there.

Uh, because one question I have about that, and this is always the debate, is like, okay, if you have a— if you're GPU poor and using your own rig, um, there's some fair amount of debate of like, does NVLink— how much does NVLink, the, the lack of NVLink, destroy performance? And like, if you try to Google for this topic, there's very like Far information, there's like, there's like one Hugging Face page where like Sylvain did an experiment one time, and Sylvain and maybe even Stas showed like some 25% degradation, but then there's some debate about, okay, that's, that's the right number and whatever. And then like there's Tim Detmers' blog who just flatly says that you don't need it. And then there's like other people are like, you do need it, uh, or like it does like make a huge difference. So I'm just like curious if you like— yeah, I don't, I don't know the answer either.

It's NVLink doesn't work for 30— for 4090s, and I believe maybe barely works for 3090s. I forget, but, um, that's true.

Yeah, so if you're GPU poor, probably— yeah, yeah, yeah, yeah, it's like you're stuck.

Yeah, yeah, it's only 3090s, which is why for a while 3090s were sort of the golden child. I don't have 3090s. I could find 4090s before I could find 3090s. That's how insane it is. Um, so the exact numbers on stuff like this is like, you know, Tim Detmers talked about how consumer cards are throttled a bit on purpose with their drivers. Um, and NVIDIA knows about this. And so when I was doing the math recently to get like a new A4500 at the same VRAM as the 4090 was an extra $500, I think, $500 or $600. So it was like a tier up because you're already paying, you know, $1,800 for a— but it was about the same-ish. I think it was like $2,400 or $2,500 for the A4500. If I had to build my rig again, I would probably buy the card.

You would probably buy what card?

Sorry, I buy the A4500 because it is a thing where like more than the performance hit, the power usage and the heat and everything. Yeah, and the space, it's a fraction. Yeah, right. Like I'm, I have two water-cooled 4090s and I have to have a giant case because of it versus it, I could fit four A4500s in that case, I'm pretty sure. And because I think they're single slot, yeah. If not, they're too small.

Yeah, they're very nice and slim.

Yeah, they're super slim and they use like a fourth of the wattage. So, and also they have more CUDA cores available to them for actually to be used by for training. However, if you can't find them, which a lot of people can't, you can get by, right? I've done my job. Granted my job has had limited training, but I've done my job with just two 4090s to do 4 iterations on LoRA tuning, uh, Llama 38 billion. It's about somewhere between 4 to 6 hours to get full training done. Yeah, um, it's—

I think it's a pretty good card. Yeah, it's a pretty nice-ish card for sure. Like Yeah, it's like small and yeah, makes sense.

It works. And it's, you know, it's one of those things where it's like, if you can find it, I'd probably do it. But if not, like, you'll be okay. It sucks because NVIDIA is sort of squeezing us out, but yeah, we'll find a way.

They're waging a war on the GPU cores.

It's great. We love it really bad.

Um, I like having the rig though.

Oh, it's great.

Still, for some reason, like, I just— yeah, it's kind of crazy. It's 2024, I still like having a rig, but still like having a rig.

Well, it's because you own it. You own it.

I don't know, it's just like, it's still slight— little bit easier for me to like tinker around. Yeah.

And you know, you never get bored if you just sit there and upgrade computer parts every once in a while. Gives you a fun side hobby to do while you stare at the pretty fans going RGB or not. Um, let's see, we want to know if fine-tuning can ever catch up to frontier models. Oh, that's a fun question. It's—

well, if you look at Technium's models, he has his community fine-tunes and they like exceed the, the base on many benchmarks because he tries to fine-tune them broadly. It's almost like a continued pre-training in a way, the— from what he's doing.

Exactly. Um, and so it's, it's largely on the community, right? And so if we can go through and make those fine-tunes and make them competitive, that's great. In some ways, we're also fighting a losing war because data, right? Because the closed-source people have access to a ton more data than we do. But you know, as you said, Hamel Husain, we're kind of keeping pace as best we can, you know. Um, also notice this in the Discord, someone getting excited because they never heard about the A4500s before and the price, how it looks affordable because it's close to the 3090. Make sure you're looking at the right one. The older version is 20GB, the new version is 24GB. Either or, again, is fine in my opinion. I almost went with the 20 gigs instead of the 24 gigs, and I believe Sam actually recommended the 20 gigs for me instead of 4090s back in the day when I was building that. But here we are. I had to do some debugging with FP8, so I needed 4090s.

Wait, sorry, the A40—

the—

oh, the 4090s do have FP8, but the A4500s don't because they're Ampere series.

Okay, yeah, the prior ones.

There's A4500s that are ADAs and they didn't make it an H? Nope. You hate to see it.

So, um, but that's the newer version, that's the 24 gig that has the ADA capability. What are some ways to ensure that prompting and tokens work at inference time? That's a fun one. I was just looking at that. The best thing I found to do is upload your model to the hub or just have it locally and load it in a pipeline. And do— I believe it's model.generate or even just calling the pipeline itself. Phenomenal guide, which I will link here. Well, I guess I can't link it there. I'll link it in the Discord that talks about chat templating. And it was the exact debugging guide I used when I was looking at trying to find problems with inference when it came to taking these trained models and trying to look at how are my tokens looking. Uh, let's see.

Yeah, I mean, the, the Hugging Face chat templates are pretty cool, but I don't— I guess like, I actually don't understand, like, can you use those directly in Axolotl? I'm not sure.

No, you can't. That was actually a big thing I was talking about with Wing because Axlol doesn't do that. And so you essentially have to copy and paste your entire chat template with the question you want to answer, uh, which was very concerning for me because I felt like I was doing something wrong because I felt like that should be magically happening.

Yeah, it does feel like the right design pattern in the future is like maybe standardizing on something like that, like the Hugging Face chat template, which I think you can— maybe that's what you're advocating for in the Discord.

Yeah, I was talking with him about that and it's like, that'd be a good thing at some point that even I might look at doing. 'Cause that's, uh, you know, when you merge your PEFT model, one thing I immediately wanted to do was test the weights to make sure that everything worked. So that's huge.

Yeah. And like, yeah, if you could do that, it would actually get rid of lots of spaghetti code. 'Cause like a lot of the spaghetti code is around the prompt templating craziness. Uh, but the masking part, I don't know, the input/output masking. Okay, maybe you can't, maybe it's like, yeah.

Um, what do I need to do to run inference on an 8 billion model using a single NVIDIA 4090? My current calculations show a minimum GPU requirement of 13 gigs. That's about right if you're doing half precision. You can do quantize. You can also get a bit slow and do offloading to offload onto the CPU. It'll be a bit slower, but you can run it. And that's just do device_map equals auto whenever you're bringing the model in from Transformers.

Yeah, I mean, like a user experience, standpoint, like, sweet spot is, I would say, VLLM, use AutoAWQ or some quantization method like that. It's a nice, you know, you can always like get faster with TRTLM, but then you have to like suffer.

I quickly fell in love with VLLM. That's what the StarCoder 2 Instruct people used for their dataset generation. It was like, holy crap, this is easy to use. You just point to the folder and it goes.

Yeah, for sure.

Why aren't models trained in int8? That's a good question. Because it's very unstable. NVIDIA's come out with something called Transformers Engine, or— oh, there's another one that I can't remember off the top of my head. MSAMP. And they are experimental. You can train in 8-bit. Practically, from what I've heard, just around the bend, is people have tried it, they still go back to BF16. It's just better. That's part of why I'm doing this Llama 38 billion experiment is I want to go see it for myself. But that's just what I've heard. I know PyTorch though is adding official support in. And so maybe they figured out some secrets, secret sauce with quantization on the fly plus native FP8. I don't know. I haven't looked too much into that repo yet, but I'm hoping that'll change a bit in the future. For right now, it's just not realistic from a quality perspective.

And that's for— you were answering that both for Int8 and for FP8, because you mentioned the Transformer engine, which is FP8, not Int8, right?

Yeah, it's one of those weird things, right, where it's— int8 for quantization or FP8. Um, so I was answering for FP8, AKA native 8-bit.

Got it.

Um, and so yeah, I've also heard what, like, reports from people about instability and difficulty using FP— oh, like, yeah, using FP8, which is why it's like kind of surprising that they doubled down with FP4 with the Blackwell architecture, right? Like, do you have any thoughts about that?

Yeah, I have thoughts. How much of that I can say, I don't know. But, um, it's, it's definitely interesting because clearly they have some sort of special sauce that lets them train models decently well, because otherwise they wouldn't be doing it. The thing that's weird is we can't necessarily recreate it, you know, because otherwise everyone would be doing FP8 if it was truly as good as it was. So there's some other stuff happening on the back end. I would love to know when the Llama 3 paper finally comes out what they trained it, because they have, you know, all those H100s which are optimized for FP8. And so I would love to know if they actually were able to train it in FP8 well, or if it wound up having to be, you know, BF16, which that's a bit of a misnomer. It's BF16 with FP8 essentially. Hmm. So yeah, it's definitely a wait and see sort of question.

Aman wants to know if you can explain the difference between RLHF, DPO, and SFT.

I cannot. I have not touched that library at all, sadly. But I think maybe one of my coworkers will be here that talks about it. I'll double-check the list. But at the very least, I know nothing about that. I'm sorry.

I don't know too much about that either, to be honest.

What have you seen in terms of reasonable scale in applications and projects where Accelerate failed and what was missing that forced you to change or update Accelerate? Not a lot. So in the presentation, I mentioned how Lucid Rains uses Accelerate for most of his projects. He— what he does is he recreates closed source papers and trainings and recrea— and fully recreates them from scratch. And he's orchestrating these on trainings of over 1,000 GPUs. Accelerate can't fail per se, because if Accelerate fails, all we are is a light wrapper around what everyone else is orchestrating. So it'd be very rare for Accelerate to fail in a few ways. The only thing I think I might have seen is occasionally like weird timeout things and timeout issues that I still haven't diagnosed the cause of, and I'm still not 100% sure if it's Accelerate. Because training in just distributed is just hard. You get weird issues that you don't know the answer to that you got to go figure out. Your best advice is the experts at PyTorch, if not NVIDIA engineers. So that's a good question.

There's one I thought was interesting and I'd like to hear your take on it. Zach. Um, uh, are the Chinchilla scaling laws still relevant, and can we use them or other scaling laws to estimate which models will benefit the most from additional training?

That's a fun one because I've been mildly keeping an eye on that space. So for a while we thought yes, right? Chinchilla scaling laws are sort of the end-all be-all. They still kind of are, except what Llama 3 showed us and what all these other labs are showing us as well is you can still just keep training. You can still just keep training and do good, but I still think the scaling laws matter.

Yeah, I like the scaling laws tell you how to optimally allocate a fixed number of flops between parameters and data. And like for that, they are still like the chinchilla. There's been like revisions and some window dressing around Chinchilla, but like for that, they're still correct, but they do not tell you how to get the best model because the answer to get the best model is always to just continue training until it has converged. Right. And so I, but I think what's maybe interesting for people who are doing continued pre-training and fine-tuning is that some empirical evidence and most people's intuition is that models that are undertrained, like chinchilla models or the, like, original Hoffman scaling law models, um, and like anything but Llama 3, kind of, uh, like models that are heavily undertrained should be more steerable and easier to fine-tune because there should be more, like, kind of slop in the weights. Um, I don't know to what extent, Zach, that has been your experience.

Uh, no, I can agree with that thinking. I don't have comments on Llama 3 yet. The only comments I do know is when I was looking at my base checkpoints, because I went actually a bit over in my PEFT tuning compared to what I think you guys recommend, because usually it's like 1 to 2 iterations through the data, and I trained for 4 if not 5. The 4th or 5th one showed the best while still improving my eval loss. So at least for 8 billion, there might be some space there still. But otherwise, fully agree. Is it fair to say that TensorFlow isn't really relevant for any of this? Hugging Face Accelerate, et cetera. It seems like everything is built off PyTorch now. Yes and no. TensorFlow, and especially Keras, is still very much so a backend in Transformers. Support for it is, you know, yes, it's died down a little bit, but now we have a few people that are working directly on it. And so starting to go back up, but yes, it, the general trend for the last few years has been PyTorch started out for research and then people just started preferring that framework a lot for a lot of things. No, I'm not going to say one is better than the other because they're both equal. You know, it's just everything I do is in PyTorch. It's how I got started and I'm still in PyTorch. Kind of how it is. And I'm sure you guys feel mildly similar about that.

There was a while when it seemed like JAX was going to gain a lot of popularity and then I'd never hear people talk about it anymore. Do you still have, like, is there much happening that you see in JetX?

Kinda.

The fun with—

The danger zone. Danger zone.

Conversation. Some person's gonna come in.

God, shoot me. You're allowed to take me in the back room. It's fine. It's Google. At the end of the day, it's Google. You know, so take it with a grain of salt. They rewrite everything they do every few years. You know, it depends. It's a risk. It's a gamble. And that's sort of, I think, part of why people are hesitant to go use it. And also, I don't know. It's like with XLA. For a while, it was just on TPUs. And only researchers had access to TPUs. And so you didn't really do a whole lot with XLA unless you were a researcher. And they've only recently brought it over to GPUs. And it works. It's fine. It's just weird because I don't, wouldn't have thought that they would have moved over to GPUs. So it's kind of a situation like that of it'll be neat to see where it is in 3 years, if it's still a thing in 3 years. And if it is, then that's probably a good sign. Yep.

I think also it's, it is a like very beautiful framework for thinking about, it's like not just an auto-differentiation framework, it's like a program transformation framework. And so it's like in some galaxy brain way, it's like the right way.

To do this.

However, that makes it a lot harder to use. Like, it's, you know, like pure functional programming when you get down to it. And a lot of people who are researchers and tinkerers and data scientists are not comfortable with like managing an RNG monad, you know? And like JAX kind of like puts that stuff in your face.

You just convinced me not to use it.

I like that sort of thing, but—

Hamel Husain, some of us enjoy the nerdy stuff.

I enjoy nerdy stuff too, but I mean, sometimes, I mean, yeah, gotta pick and choose.

Uh, let's see, any other interesting ones? Any particular tips or limitations for training on Apple Silicon? That's a fun one. I can talk about this since PyTorch admitted it. Sumit tweeted about this last month. Silicon with PyTorch is okay. Inference, it's great. Training, you're looking at 3 different lower-end libraries that all try and do the same thing, each of them with varying degrees of success. One of them is PyTorch, and they have admitted that theirs is one of the worst ones. They're looking to revamp that and get more into it and actually invest time in that in the next few months, I believe Sumit said. So for right now, it's good for inference, training. I'm sorry if you can hear my cat. Perfect. I can hear the cat.

It's not a big deal.

Don't worry about it. Okay. For training, it's mixed. Take it with a grain of salt and try with an NVIDIA GPU if you have one. MacBooks are cool. I refuse to sign on to that. And so thankfully I don't have this headache.

Look, so much stuff can go wrong. I'm like, I'm just going to go the paved path here. I'm using NVIDIA GPUs, using Axolotl. Yeah. You know, just works. Something battle-tested, like, because I don't have time. Like, there's so much stuff to do. Yeah. Just like barely have time to fix the data usually. So.

Yeah, and I think it's unlikely that Apple Silicon is going to become a preferred target for training because like, there, it's like a system on a chip, CPU, GPU together. Like, there isn't something that looks like a server card with fast GPU interconnect. And then, and additionally, on top of that, you need fast communication between each server blade. Like, and that's just not something where Apple has done work. So it, like, in 10 years, we might be looking back at this the way somebody 10 years ago would've been like, Apple's not making a GPU. Um, you know, Apple's not gonna make their own hardware. It, so like, in the foreseeable future, there is not the infrastructure in place to make it a really great target for training. Um, and so, Zack, do you agree?

We can't hear.

I didn't hear what you said.

Is that— yeah.

Oh yeah, sorry, I muted my mic because Max is right here. Uh, the biggest sign of that— I agree with that. The biggest sign for me is, uh, if you're not familiar with Cash on Twitter who writes Dingboard, uh, I love him. He's phenomenal. He's hilarious but also has really good insights. He was this close to executing on like a cluster of the new Mac workstations to have the total VRAM usage. And then he said, no, actually it's much better if I just stick with NVIDIA. Did a total 180, was about to buy it all, and then went, nope, actually going with NVIDIA because everything's too unstable.

Hmm. I think, yeah, similar story with George Hotz, right? And the AMD chips. It's like, yeah.

Yes. AMD is a whole different ballgame. But yes. Uh, how would you serve multiple LoRAs with Accelerate Inference? You know, I was thinking about that, how you could like hot swap LoRAs in and out. I don't think we've done that yet. I don't necessarily know of anyone that's done that yet. It'd be a neat problem.

Does VLM not have that?

I don't know.

You can load several LoRAs.

Yeah, it does.

You can.

Oh, nice.

It has a hot swap thing. Kind of like you can go between LoRAs. It's pretty cool.

It's like a single batch or no?

I didn't try that part.

Yeah, you're slowly making me love it.

Yeah, but it's really cool actually. Like, I can pull up the docs real quick actually.

Yeah, that's fascinating because that was a thought I had, I think after the class on earlier this week, where it was like, huh, that'd be a fun experiment to see if we could go do. Yeah, someone go do that, document it, and write a blog. Tell us what happens.

I think that Lorax does that.

Okay, so you see this one? So basically using LoRa adapters, and then you can, uh, you can serve LoRa adapters. Um, just a second. Yeah, see, like these LoRA quests, you can have like different adapters. I don't know about how you would hop. Yeah, I guess like because you already— yeah, you can just like at inference in your like in the generate, you can just decide to pass in a different adapter apparently.

Did you click that multiLoRA inference.py file? Yeah, I remember like looking at this in the past and not quite figuring out whether they would be hot-swapped. Yeah, process.

Yeah, I don't know, I haven't tried this yet. Maybe it's not hot-swapped, maybe I'm just not remembering.

Yeah, we'd have to go into— because like they're sort of saying it's like as though it's a separate model, and I don't know, like separate—

I guess like you're calling— you're not, uh, so like you're loading the model separately. Yeah, you see, and then it's only in the whatever forward pass that you are then passing the adapter. So I guess that's hot swap. I might not know the— I mean, maybe I'm just like assuming what the word hot swap means. I mean, I use it mostly.

There's like versions where you need to merge it into the model weights and that's, you know, that's fine. It's nice to not have to merge it into the model weights because it's just like easier to manage the artifacts. And then the, like, the next optimization on top of that is as a batch is going through, you route some, you know, rows in the matrix through different LoRAs. And that's the, like, really cool thing about LoRAs, like, especially if you're serving, like, many people's fine-tunes. Like, you have a fine-tune for every user, you have a fine-tune for every region, whatever. It's great to just have, like, your batches just get collected together and you get most of the throughput benefit of the, of batching for the large model with all the customization of LoRAs. So that's the like full, like per-request LoRA parallelization that you'd really like. That's why it's so nice to have these parallel adapters like LoRA instead of sequential adapters, like replacing the language model head or like putting an adapter at the end of every linear layer or whatever.

Makes sense. Don't pay attention. I was just scrolling around.

Yeah. Um, somebody asked about Mixture of LoRA. I don't know if that's, um, I guess that would be like live routing between LoRAs like they were experts. I don't know if I haven't heard of that being used.

Sounds like a fun idea.

Yeah. I guess, yeah, you'd need to train them together the way you train experts together because you need a, like, routing between each one.

Well, didn't, like, what was the Kraken model that just came out that has, like, 6 different models each on completely different tasks to help route through things?

Oh God, please tell me I bookmarked You could make a really janky version of it. Like, yeah, just have a classifier or something. The routed is a completely separate model.

Half of what that was.

Isn't the MOE thing, it's like, isn't it trained to route? It's how in the network itself is routing to a different path more directly than like, you know, going outside.

Yes.

Okay. Yeah, the person asking about Mixture of LoRAs, I think, is posting about it in the Discord saying like, yeah, train LoRAs separately, get a router as a, yeah, LoRA Mixture of Experts.

Kraken LoRA. And the picture is quite literally Cthulhu with a bunch of different fingers pointing to different LoRAs that perform different tasks.

Nice.

It's a collection of experts is what they're calling it.

Oh boy.

Yeah. Combines the best Python, SQL, function calling, reasoning, and German models by applying dynamic LoRAs on runtimes so far.

Oh, I see. The top tentacle is holding some muesli, I guess. That's the German one.

Ah, that makes sense. Yeah, dynamic model routing uses a sequence classification model to route inputs to the most suitable language model based on the characteristics. Fascinating. I did drop that in the Discord in case people are getting full mouth. How do you decide a fine-tuning project? Even this Llama 38 billion experiment you're running now, what led you to say, let's do it? I read the paper. It seemed neat. It seemed like something that was reasonable for me to go and do. The steps were all right there. And it was something where I was like, okay, that's fine. It's a pet problem. Doesn't have to be unique. You could be just recreating what other people did. Like, I think that's what one of my winning solutions at a little hackathon back in college was. I was quite literally taking the ULMFiT lesson from Jeremy's Fast.ai course back in the day and applying it on news articles. And then we just served it. And that all in all took like a day and a half. And so generally when I think of ideas, it's sort of what interests me and what's relevant to what I'm doing right now and try and find a connection there, either previous work that people have done, especially if it's open source or data and just go with it. One thing I sort of like about, there's a quote from one of the guys on Mythbusters. And it basically goes, the difference between just messing around in science is writing it down. Even if you're just messing around with data, write it down. Great, now it's a blog. Model fails? Great, that's fine. Write it down. That's a lesson learned. And all it does is bolster you and your experience. Um, there's another question similarly that you guys might actually have some thoughts on. When you work on a fine-tuning project, do you typically get constraints for inference time and memory consumption when you start, or how do you choose the sweet spot of model size and performance? My first guess would be go off of your budget. That dictates how much VRAM you have available, and then that dictates what kind of model you can train, right? Because on 2 4090s, I can't train the 70 billion Llama, but I can train 8 billion and smaller. Um, and time again is cost. Is it— it's free for me to just run that for 2 days or I can pay $80 and spin up an H100 and get it done in an hour or two. But that's my thought. What are you guys' thoughts on that?

Yeah, it's similar. Like, okay, like the honeycomb example, which I'm going to do in this course, in that situation, the incumbent model was OpenAI, right? So you like look at their latency, their cost, whatever. And that's like their latency and their cost is pretty good. Like, it's hard to compete with that. Now, if you're going to replace— there's a— humans have a cognitive bias, by the way. Like, once you already have something, you value it more. Same thing with models. A lot of times, like, you're like, eh, it has to be, like, better. It has to be, like, markedly better for anyone to notice, to really notice. I mean, you can measure it and stuff, but, like, to really— in this case, like, I also do consulting, so I, like, really need it to be better to, like, you could noticeably feel it from a user experience perspective. So I want it to be like a lot faster. And so in that case, it was pretty clear to me like it needs to be a 7 billion parameter model. It needs to be like small, it needs to be fast, and I need to make it actually make the prediction quality like higher than, you know, GPT-3.5 through fine-tuning. So it was kind of like that, but usually it's like, okay, What is the smallest model I can get away with for the quality threshold, basically?

I have a bunch of thoughts on this. I mean, first of all, I generally think of training compute as free. Like we talked about—

It is after this course, yeah.

Yeah. With all the credits that you have, you will need to find ways to use compute. So in the past, you would have used 8 billion for a model, and I like, well, I got to find something to do with all these credits. Um, but inference is so vastly more expensive for all the projects I work on than training that I just think like it's not even worth thinking about training costs. Um, there's one project we're in the middle of working on. This is, uh, I've talked about this before, this multimodal alt text generation, so image to text, like we're doing the most expensive training runs that I've worked on and they still like, it's a trivial fraction of inference costs. It's not even like a meaningful fraction of the amount of time that we spend talking. Like the, we, I get paid hourly for that project. The amount of time that I spend on a single call deciding what model run would cover all the models that we train for weeks. Um, so training really doesn't matter, and inference is what matters. Um, and then the other thing is the time, like your iteration speed matters. We've talked in this course about how important data is, and because data is so important, I typically will start with an 8 billion parameter model because you can iterate faster see what's breaking faster. If you want to change later on to a larger model, um, you can. And what we do is we always start with the 7 or 8 billion parameter model because it's like a smartish size. You can train it quickly and easily on a single GPU, and it's nice to not be using distributed GPUs. Um, and then frequently we'll like say like, okay, let's try a bigger model, and then that would have some cost for inference if we end up using a bigger model. And we do that and then we say, oh, actually, it's not meaningfully better. And so we almost always end up just deploying 7 or 8 billion parameter models because they're a little faster than a 13 billion parameter model or a little cheaper. Like you'd use a less powerful GPU. Um, and my experience is that we can't actually tell the difference when you just side-by-side look at the results. So, uh, model size is like a less interesting problem than you would expect. We just always run 7 or 8 billion parameter models. Um, and it's not because we're scared to train. Like, I would train a 70 billion parameter and not worry about the cost. It just isn't worth it.

Hmm.

Let's see. Here's an interesting one. Why aren't people fine-tuning more on Phi-3? Is it because it's smaller, or is it just Mistral small enough with Flora training? I think Phi series, I don't know. There's something weird with the data and how they trained it that isn't showing real-world performance. From what I've gathered on Twitter, and that's generally why people don't like it. Like, to be biased, I generally make my decisions on what models I look at based on Anton and sort of his experiments, because he just sits there and hacks with prompts all day and figures out what he wants to run locally. And for like, I think 12 hours, he said Phi-3 was actually good, and then he threw it into his real-world scenarios and it didn't perform. Whereas Llama 70 billion performed out of the box phenomenal. 8 billion is still good for him if he wants to do a small fine-tune. So it's just the Phi series is weird, I think, would be an apt way of saying that. I guess it'd be cool if it worked, but it's kind of the situation, I think, where 7 or 8 billion is kind of the threshold of where things need to be at for things to happen still, parameter-wise.

So you might be on time, right? Yes.

Yeah.

Yeah.

Um, I found how to export unanswered questions, so I will export the CSV, drop it into the chat for, um, for this session.

That's a great homework for someone to build some kind of RAG application or whatever you want to do.

Nice.

Yeah. So if we want to come back, we have questions both answered and unanswered saved that I'll share with everyone. And we'll share this recording, of course.