Napkin Math For Fine Tuning Pt. 2 w/Johno Whitaker

would have meant less energy, but they both just had a ton of energy. People really like them.

All right. Sorry, I'm a couple minutes late. I got totally caught up typing a longer response to Dan's question about state-space models and like didn't realize we had actually gotten to the time. All right. Hello everybody. Let me just catch up on the messages.

All right.

So we're on Part 2, the follow-up that we didn't know we were going to do. Um, take a deep breath, I've brewed something non-caffeinated. We're going to go a little slower, um, we're going to linger on some definitions a little bit more. There was things that were confusing in the first one, um, hopefully this is the time to address that, to rectify any things I said that were incorrect. Um, if you two other panelists spot anything maybe from the last one where you're like, oh, you know, he said This, and he probably actually meant this. Yeah, we can correct all of those. Does that sound good?

Sounds good.

All right. Do either of you have questions that you think we should start on that you'd seen come up that maybe I haven't addressed?

Yeah. There's one that I liked. It's like, do you always try to saturate your GPU completely, like increase the batch size as much as possible? What do you do? Like, what's kind of, how do you think about it?

Right. So this is one that I had to, I say recently, like maybe last year, realized I wasn't quite correct. My assumption was larger batch size is always better because, you know, then we're saturating the GPU memory and things like that. It should be good. And that is largely true. Like if you are doing a batch size 2 and you could be doing a batch size 4, especially if you're memory bandwidth bound, like a lot of us are, then go to batch size 4, right? Because you're now, especially if you're copying weights from the CPU or anything like that, or you're unsharding sharded weights, having to do that fewer times is going to be a good thing. The only case where the question becomes a little nuanced, and this is a mistake that I made, I was like very proud to see my memory utilization at 98%, you know, like, oh, I'm like flying right by the wire. And what I realized in some testing with Jeremy was if, say, you could have a batch size of 30 or 40 and you're more compute bound, say, but the batch size of 40 is like 99% memory utilization, then by not having that extra space for cache, it actually slows down PyTorch, especially in some weird cases. I don't know that I can like point to where, but we saw a definite slowdown. Like it was actually slower to go 99% versus 90%. So having a little bit of—

How much slower? Like just—

Like 40% slower or something. Like it's significant. That's a lot. That's a lot. But that was only when it was like right at the edge, right? As soon as you backed off, it was like, okay, cool. Now it's able to like pre-cache the next layer and now it's not having to like pause computation while it copies something in because there wasn't room in the cache. So that's really like, that's really the thing is like having enough headroom, especially in distributed stuff for pre-caching the next layer, like uncharting and pre-caching the next layer. That's, that's nice. Um, so yeah, that's what I would say now is like go to a larger comfortable batch size. The other reason is I've done this too often, 99% memory utilization, and then after 200 steps, like 2 hours of training, you do your evals, right? And the evals, you know, has a different batch size or something like that, or you hadn't cleared up memory, and then you get out of memory and your whole run crashes, and now you've wasted that time. Whereas if you're at 78% memory utilization, batch size of 8, then you come to your evals and something is a little different, or you loading it, it doesn't matter, you know, it bumps up a little bit and down and you're fine. So giving yourself that little bit of extra wiggle room is nice. But if you can, like the largest batch size that you can comfortably get away with without feeling like, you know, all the gauges are in the red, that tends to be a good practice.

If you primarily were just interested in getting a job done as opposed to saving compute, how do you think about the trade-off between adding more GPUs, which will obviously make it run faster and maybe even or if you were at a fixed budget, maybe more GPUs, but they were less expensive versus fewer GPUs or one GPU, so that you probably have less total compute, but then you have less I/O as well.

So one part of this is something that you, you brought up, uh, I think in a talk the other day, which is, um, if your time is worth anything, and the memory savings are going to take a bunch of time, or even the benchmarks are going to take a bunch of time, you know, then, then yeah, at some point, like, you're spending $2 on GPUs instead of $4, and you're spending $200 of, of your time. So that's, that's definitely a consideration. Um, but then in terms of like the actual, um, yeah, okay, assuming that it's easy enough to try any of these different things, what should I be going for? One thing that I found quite interesting when I was running these experiments was we'd done QLoRA with FSDP. It's like, okay, cool, we can use fewer GPUs or we can use cheaper GPUs and we can spread the load out over 8 3090s. This is so much cheaper than 8 H100s. Obviously the GPUs are slower, but like, because we can do all this memory saving, this should be good. And what we realized, let me scroll down. Yeah, here's the money shot. It turns out like the pricing of GPUs on cloud providers is the pricing for a reason. Like if you can get twice as much done on a GPU, then even though that GPU is twice as expensive, like the total cost ends up about the same. So this ballpark cost, like I was expecting a huge variance, right? Because oh wait, I'm training on 4 H100s. You know, this is like $12 an hour or something like that. That's crazy. But then it takes 600 seconds, you know, versus, oh, I'm training on like 4 A5000s, much smaller, much cheaper GPUs. It takes like 10 times as long or 5 times as long. And so even though it's much cheaper, like, well, it's cheaper, but it takes longer. The costs actually balance out quite nicely, you know. Doubling the number of GPUs ideally doubles the speed because I can fit, you know, a batch on each one and they're going, that's great. But it doubles the cost-ish. Yeah, so that was an interesting case study for me to realize, like, oh, there's a reason people are willing to pay that much more per hour for H100s versus A100s, and it's because they can get, you know, some multiple of their training speed. They can double their training speed or whatever the case is. And so yeah, that kind of freed me because it's like, okay, I'm often willing to wait overnight for something. It's a one-off experiment or I'm generating some data. I have two 3090s at home. You know, I can just run it on there. But I want to test, like I want to do some ablations. I want to test DORA versus LoRA and I want the results out, you know, ASAP. Like cool, I'll spin up a 4 or 8 GPU node. And as long as I'm not wasting too much time on on setup and forgetting to turn it off and all of those caveats, you know, that's 4 or 8 times faster than on my home machine. That's really nice. And that's, it's cool that we can just exchange more money for more compute and more compute means it goes faster and faster means actually the total cost is, is not that much more.

I think implicit in that is you're saying the I/O cost actually is not big enough to matter. So like scaling up Scaling to more GPUs doesn't ever, like it scales roughly linearly.

Yeah. As soon as you go like multi-node, right? So now I've got 8 GPUs on one machine here and 8 GPUs on another. So this is a little tip if you're doing FSDP, there's a full shard versus hybrid shard. Like I'm going to assume that you can fit your model on at least one 8-GPU machine. So then what you shouldn't be doing is sharding everything across the nodes because there's going to be so much talk over what is like a network link. And unless you're in a fancy data center with fiber optics, that's going to be really slow. So you want to do data parallel across the nodes and then within the node you can do your other tricks. And then it does, it does scale pretty close to linear. You know, people are, it's tricky and it was very difficult to get good utilization, but these days people are getting better and better at figuring out ways to scale to tens, hundreds, thousands of GPUs without too much of a penalty.

I try really hard not to go to multi-node.

Yeah, no, I— same, you know, if like, just, yeah, again, like, as long as I'm willing to wait twice as long on a single 8-GPU node, like, that's great. If I can do it on a 2-GPU node, even better. If I can do it on my home machine, even better, because there's no setup time. And yeah, I still, I mean, we have funding now. I work at a company. It's not just me paying the bills, but I still have this thing of like every second that this H100 machine is sitting there, like not training, you know, like I'm downloading the data or trying to figure out some, it's like my brain is just going, you're just wasting money. You know, it's like I can't get out of that mindset.

How do you think about hyperparameter tuning? Like, do you do a lot of hyperparameter tuning or do you kind of have like a kind of a thing that you kind of go to? You kind of know what you like or something and you just go with it?

It depends. It's been a little while since I've had a task where I really needed to optimize. Like, for something like Kaggle, you really want to optimize that final few percent of performance. And then you really want to spend the time tweaking to get 1 or 2% increase. What I found with a lot of the things recently is either you can find a model that does it zero-shot. There's no fine-tuning. In fact, there's like a joke that I'm— every time someone sees a thing where fine-tuning works, they say, look, Johno, why are you always saying what's the point of fine-tuning? Fine-tuning is not going to work.

I'm going to ask you about that next, by the way.

Yeah. So I've somehow become that skeptic. But then if it does work, it's like, well, the next stage is pretty much anything works. I mean, I hate to say that because I really love hyper-optimizing everything. But like LoRA with the default settings that you'll find in one of Wing's configs, right, like rank 32, learning rate 2e minus, whatever the default is, it mostly learns what you want it to learn, if it's learnable at all, right? And if it's impossible to learn, then tweaking your hyperparameters isn't going to fix it. And if it is learnable, then great, like maybe picking DORA versus LoRA will get you a few extra percent or getting the learning rate just so will get you a few extra percent. But mostly at the moment, like I'm not in some product where 78 to 79% accuracy matters. It's more like testing out ideas, is this possible at all? And then, yeah, I'm not spending time running ablations with 18 different learning rates. I'm just picking something that seems like it's learning okay. I mean, it's not going crazy. It's what everyone else uses. And that's fine.

So, okay.

Yeah.

So one, the next thing I'm going to ask you, which I already kind of gave you a hint. So like you're working at Answer AI, you're doing lots of research and also development, the D part. The development is like, Okay, make lots of useful things with AI, maybe some software, maybe some user-facing applications, things like that. Do you suspect that you will be fine-tuning a lot when building those kind of applications? What's your intuition, even if it's your own opinion or whatever? Like, do you think that you will be spending a lot of time fine-tuning? You'd be spending a lot of your time the building the applications part of that, or how does it fit in? Or, and then also maybe like, oh yeah, like tell me about your skepticism as well.

Yeah. So it seems like if you're building products or applications on top of language models, what you want is to be able to customize them, right? I have an off-the-shelf model, but it doesn't know that it's summarizing, it's supposed to summarize papers. It's like a general purpose thing. So you can customize it with a prompt. You can say, hey, you're a helpful archive paper summarizer, right? And for a lot of things, that's enough. But then sometimes when you want to customize it, it's like, well, I don't just want it to talk a certain way. I want it to always follow a certain format, right? Or something like that. So now you've shown us good examples of fine-tuning being perfect for that. I'd like the model to always produce SQL and only SQL. And I'd like it to use the conventions that my training data has, right? And so that's like a stylistic formatting thing that is perfect for fine-tuning. And if you have a fixed set of tools, fine-tuning on those fixed set of tools, great. There's other ways to customize it. And so we started out with thinking like, if we can get fine-tuning efficient enough that it can run on someone's home machine and it can be run effectively for free continuously, there's a lot of products that maybe are unlocked by that. So we spent and are spending a lot of time. There's a couple of people who are having all sorts of fun with extreme quantization and really efficient kernels and all the stuff to make it possible to hypothetically fine-tune in all these different places all the time. But that's not the only option. And so another one is to put a lot of data in your context, right? And this is becoming easier and easier. When you had a 4,000-token context, Maybe that's not enough, but there's a lot of cases where fine-tuning it might do the right style, but what I actually want is direct copy and pasting from a reference document. And in that case, having the reference document in your context makes a lot of sense. And so this is like a very appealing thing. Now there's a trade-off, right? Longer context takes longer time to process when you're doing it. And so you might think, well, is it worth it if it's going to take 30 seconds to load up all my code and all my reference documentation into my assistant before the user gets a response? Well, up along comes, you know, Google next couple of days are going to announce their like prefix caching, right? Where I say, look, I always want my conversation to start with all of the docs of my library, all of the code examples, the actual source code, a bunch of like other preamble and useful references. And I'm going to compute all the activations I can from that, all the keys and values and all the different layers, everything I possibly can compute ahead of time. I'll compute that and I'll keep it in like VRAM, you know, I'll keep it warm. And then when a user asks a query, all I'm doing is continuing generation. I've already got my KB cache warm. And now I can like copy and paste with the full power of attention. No need for like lossy retrieval, no need for like fine-tuning and hoping that it remembers the exact thing. No, no, it copies and pastes with the attention mechanism that's good at that directly from the context. So yeah, there's a lot of different options now. I think there'll be some space for fine-tuning. I don't like, I'm not fine-tuning is dead. Although I'm looking forward to that talk.

Me too.

But I do also feel that maybe people should remember that you only go to fine-tuning when these other options aren't right, you know, and there's going to be different things.

There's a question in the Q&A that's a jump from what we've been talking about, but I'm quite curious. I don't know if you have worked on TPUs or thought about TPUs much at all. But there's a question. Can we briefly talk about TPUs and how they fit into the napkin math? And if not TPUs, like other, I guess maybe TPUs are unique, but like other accelerators that are not GPU-like.

Yeah. So I have not trained anything big on TPUs. And I also know that since I looked at them, there's been several new versions. But you can think of TPUs, it's very similar to the GPU math you'll be doing, right? There's a certain amount of available memory. A lot of it is like, if you have a TPU pod, they're all going to be able to access a lot of shared memory together at really high speeds, right? To the point where you can almost think of it as like just this big synchronous block of 400 gigabytes of memory or whatever. And so, and then they're often like for Google, they're in data centers where the inter-pod links are also like insanely fast. Like it's probably faster between nodes there than it is between my 3090, which is dangling out of the case, like propped up on a toilet roll with a riser, you know, you know, like that speed of my GPU to my other GPU in that same box is probably slower than their interconnect. So it's very interesting to see like dedicated hardware like that. And I think some maybe a similar analogy is like these, you know, NVIDIA things now where it's, it's like, it's still GPUs, but you kind of treat them as a unit. It's like a block of 8 GPUs that all have like shortest physical path possible to this ultra high bandwidth memory, like as, as fast as it can possibly be. They have like amazing networking to others in the stack. It's getting more to this place where it's like, my GPU is now like a giant GPU. But it's sort of in these weird little blocks, but you almost don't have to think about that. It's just like, I have a terabyte of VRAM. I'm no longer worrying about CPU offloading, you know, like that's one of the things with even someone like the PyTorch team. It's like, oh, do we really need gradient checkpointing and CPU offloading and all these things? When you're training on, you know, these amazing fast interconnect big machines with 80 gigs or 160 gigs of RAM each, and then they're all together, those are not your considerations. It's only when you're 12GB, 24GB gamer cards and last generation hardware that you really are thinking about these things. So yeah, different trade-offs, but still ultimately, like someone at Google is thinking like, okay, how fast can we copy the weights to the GPUs that need them? And if we can reduce that, that's great. But then also how long does it take to do those computations? And those two are kind of, it sort of trades off which one is your bottleneck.

Cool.

Someone asked for a link to the Google prefetching post. I don't know if it's out yet. It's like one of those, like, they're anytime now, it should be available. So go look at Logan on Twitter and see what he says. I don't have a link to it offhand, but it should be coming pretty soon for at least the Gemini Flash and Pro models. Yeah, and that's cool stuff.

Um, we got one question here from Sean Bergman that has 3 upvotes, 4 upvotes.

Nice.

Uh, I was surprised how much VRAM Llama 3.7B LoRA consumed in my 3090 GPU using the Axolotl example. I've been wondering, is there a better place to start for, uh, and I think better place to start might be better than the Axolotl example for optimizing the context length optimizer batch size and accumulation for a 7B model in a 3090. It's probably easier if you're actually looking at it because there are a bunch of pieces to that. Some of those are things you can control, some don't affect memory. So it's probably easier if you look at it and you can respond.

Let me pull up the Llama 3 example in Axolotl. I'm just going to look at lora8b.yaml. Okay, so actually pull that here. Llama 3 8B is also interesting because, you know, like, oh, it was marketed as a 7 billion parameter model. And it was It was like, oh, this is exactly the same as Llama 2 or Mistral or whatever. And then when we try to train it places where we could get a batch size of 8 before, we could barely fit a batch size of 2 with Llama. Oh wait, this is, yeah, yeah, yeah, Llama 3 8B. And so part of that, it has a much larger vocabulary. And one of the things you'll notice is that we usually quantize all the actual, like the, there's 32 Llama decoder layers, right? The transformer layers. But often we skip the embedding layer and the language modeling head. And then we need to trace the gradients back through the language modeling head. And so if it has 128,000 outputs rather than 32,000 outputs, you know, it's just that, that language modeling head has a lot more weights. Likewise, the embedding layer has a lot more weights. We often keep those in full precision. And so yeah, that, that can suck up a bit more memory and you'll see like a larger spike, you know, like the spike right at the top of that graph that shows the like ultimate peak from that final cross-entropy loss. I know the Unsloth guys, Daniel has been working on, like he's got a memory efficient cross-entropy. He's working on everything, that guy. It's pretty amazing. Okay. So this is the Axolotl config. I've never seen this before. One obvious trick would be to go straight from LoRA to fewLoRA. So load in 4-bit rather than 8-bit. Oh, right. I guess maybe LoRA 8B was 8-bit. Yeah, that's probably like, that's probably a big bang for buck thing because you're going to get suddenly half as much or a quarter as much usage for the base weights. You could try tweaking the rank to be lower rank, but that's not going to make a huge difference. I wouldn't worry about that just yet. Okay, so this is batch size of 2 per device. I guess assuming this is one GPU and then 4 gradient accumulation steps. So effective batch size of 8. Let me look at the question again. There's something about like they were asking about—

There's optimizer.

Oh yeah.

Yeah.

I mean, you know, with LoRA changing the optimizer from Adam to SGD is not going to get you a huge savings because the LoRA parameters are so much smaller than the base weights. If you were like, if you were fine-tuning the full model, then switching to SGD might be a good trick. If you're struggling, you know, and short on memory. Yeah, gradient checkpointing on, that's good. That's what it should be. I would say like, yeah, try QLoRA. You can probably get still pretty decent performance and save a lot of memory. And then other than that, like if you want longer sequences, you're just going to have to go smaller batch size and then either use more gradient accumulation steps or maybe just like lower your learning rate a little bit to account for the smaller batch size.

Actually, I'm gonna, on sequence length, I've always been surprised to see this as a parameter. My mental model, which could be wrong, is one where so much has been learned to get the current, like, EIN value stored matrices that, like, especially one of these questions, like, what is something you can do to shrink this? Like, I would have thought that if you said I want to shrink memory usage by shrinking the context length. I don't know, like, what that actually means in terms of what you cut off from those matrices. And I would have had an expectation that it depends on the model and, like, what they're doing for, like, are they doing RoPE or, like, what they're doing. But frequently, just, like, being like, hey, I'm just going to put a higher number here, I would expect that typically not to work.

So, so maybe a way to think about this is there's no there's no weights in the model that are like, these weights are for position 1 and these weights are for position 2 and these weights are for position 3. It's like each token gets processed through the same set of weights. And so when you're doing inference, for example, you feed in the string, tell me a joke, right? That's 4 tokens plus a beginning or something like that. So your sequence length there is 5, and each of those tokens goes through the different parts of the model. They interact with each other, and then you get out a set of predictions, one for each token. And you look at the one for the last token, and that tells you probabilities. And you can say, what's the highest probability? The highest probability is knock, you know. And then if you run, feed that whole sequence through, the next highest probability again, knock, you know, question mark. And so when you're doing inference, you're always having a different sequence length, sequence length 5, then 6, then 7, then 8, then 9, then 10. So likewise, during training, like often, you know, when you see the sequence length or context length setting, it's often like a max. And depending on the settings, it might not even be being fed that much, right? So what we do is we take a batch of data. These are 5 different instructions and responses. They all have different lengths. And so usually what we'll do is just add some padding to make them all the same length. And this is just because GPUs are very friendly to tensors that are, you know, even sized. Slowly we're getting support for ragged tensors and different shapes, but we pad them out so they're all the same length. So if the longest one was 800 tokens and the others were 700, 600, 712, we'd pad them all to be 800. We'd feed them through, but we'd ignore the outputs for the padding. And then we'd look at our results. So this is a big thing. Like I've seen people, if you're testing memory usage, you you're training on Alpaca and you say, look, I changed my context length from 4,000 to 8,000 and it didn't really change my memory usage. It's like, well, most of those samples are under 500 tokens in length. So when you're picking a batch, you get one that's 600 long and one that's 500 long. You get a 600-long sequence that you feed through your model. And then you get bitten because the second to last example just happens to be longer and you run out of memory. So there's various strategies that people do. One is if I have lots of really short sequences and I do have the longest context length set, I might like stack them next to each other. And then I'll just mess with the attention so that they don't interfere with each other. Yeah, another is like I always pad to the maximum length. That's just wasting a lot of compute, but whatever. Yeah, or you make sure that your largest batch with the longest sequences goes first, just so you catch any memory issues early. But yeah, this is a, it's like an easy mistake to make. And that's why, for example, in my notebook, I generate synthetic data that is like guaranteed to be 1,000 tokens. If I wanted to test a sequence length of 1,000.

And okay, and then if you said, it might mean I don't care about quality, but if you said I want to see if reducing the context length saves memory, maybe the assumption about how Axolotl would implement that is sometimes it would actually just cut off your data and you'd be okay with that.

Yeah, and that would probably be fine, right? Like, as long as, well, so often when we're training on instruction data, we feed the whole sequence through and there's an instruction and a response, but we only calculate the loss on the response.

Yep.

So if your instruction is 600 tokens and then your response is 400 tokens, but you're artificially truncating this down to only be 200 tokens long, you're not actually going to get any signal at all.

Yep.

So that's like the small caveat. But as long as there's something in there that you're training on, having the end cutoff is okay because each word before that, it's learning to predict the next word. And as long as there's some examples where it learns to predict, okay, I'm done talking now, you should be fine.

Yep.

Cool. There's a question I see. If you can go over the steps, pointers to carry out quick iterations when working with these models. One you mentioned was start with the smallest model. Yeah. What pointers can you give us? Yeah. I mean, this is always, it's always nice to have a quick iteration loop, right? So before you're trying to spin up a cloud machine and download the Llama 70B weights, which takes a while and test all that stuff, it's nice to try. So Tiny Llama is a model that's 1 billion parameters. You can run that on I think probably even a 12GB GPU. So that's like, get your code sort of working on that. Maybe move up to like a 7B parameter model. Check your evals, you know, can I like evaluate this to see how it's doing? Like, do I have like, am I measuring the things I care about? Look at the data going in, look at the predictions coming out. Like, do they seem sensible? So it's nice to like test all of those things at the lowest stage you can before you then like committing time to trying to work with these, these really big ones. Yeah, so I guess that's like, yeah, it's very generic advice, but starting small and moving up. And also being able to test, like I'll often do training runs on 200 samples or something like that, right? Is the loss going down? Am I getting errors? Like, what does the data look like? And then I can do the 20-minute training run and then the 2-hour training run. But like just having some way of like quickly testing and not needing to wait 4 hours to see results. Like ideally you get results in seconds or at most minutes to see I've made a change, you know, is it still working? Is it crashing? Yeah.

Cool.

Let's see, we've got questions on rope scaling, we've got questions on favorite tools. I'm somewhat tool agnostic, except that I like to be able to find what the actual code I'm running is doing. And so I really like being able to use something off the shelf and get a result. I've used Axolotl, I've used Platypus, had a really nice training thing. I'm sure AutoTrain is fine. I've used the Hugging Face trainer. The issue with those is that at some point you say, okay, what's like, what data is actually going into my model? Or there's something you're trying to debug. And then from a YAML file that's running something on Modal to Axolotl, which is setting up things to the Hugging Face trainer, which is then calling like some generic model class, which is then calling the actual model layer. You know, there's such a lot of abstraction there that I do quite like having something that's as close to pure PyTorch as possible. Maybe like using something like the GPT Fast implementation of a model and a simple training loop or We've got a library we worked on for the QLoRA FSDP experiments that is a starting point that I'll often use now. I wouldn't necessarily say to someone else, you should go and use that because it's now quite complicated. But when I started writing it, it was very simple. And I've kind of at least been able to piggyback as the complexity was added. Yeah. So it's kind of a funny one. If you just want to train a model and as the rest of this course has shown, the important things are the data and the evaluation. It's not like, is your code 5% faster because you're spending $1.78 versus $1.81. And actually all the value is in the other steps. So then use Axolotl like someone showed you, use a known example. But if you are wanting to do research and you are wanting to inspect things, then having a training loop that you know and you can at least dig down to the bits that you care about, that's when that starts to get very useful.

Can I pick a question from Ben when I ask you this? Uh, I'm going to change it slightly. So is there something similar to CPU RAM paging that's for GPUs? Um, like right now many of us will observe that we run out of RAM and then we get an out-of-memory error. And I'm going to tweak the question slightly, like why is it that the PyTorch hasn't built something in, or some lower-level tool hasn't built something in, so that when we're about to run out of VRAM, instead it just gets paged to CPU, the CPU RAM.

So one thing is it actually does exist. If you use Windows and the Windows Subsystem for Linux, what you'll notice is you set a training run going, like I have a 24 gig, well, I have multiple now, but when I was doing this, I had one 24 gigabyte card. I'd set some training going and the memory usage would go 23, 24, 25, 26, 27, 28. and it would be spilling over into my CPU RAM. So it's possible to do this. It's just a lot slower. Like, your CPU to GPU, I'm going to be off, my numbers might be wrong, but it's a difference between like a terabyte, sorry, a gigabyte a second versus 20 gigabytes a second. Right. And so, yeah, I think you'd almost rather try and work within those constraints. And if you're going to be offloading to the CPU, like explicitly be choosing to use CPU offloading, which is like manually doing that. And also, yeah, it's very tricky, especially then, like, if I'm renting a GPU on a shared machine, and suddenly I'm using someone else's CPU RAM, you know, like, I think there's a reason why this is not the standard. But yeah, it is possible. It's just that I don't, yeah, people don't do it. And I think the reason is the CPU to GPU RAM speed is just a lot slower. And so if you're going to be doing that, fine, but do it and be aware of it rather than hoping that PyTorch magically does it. Yeah, any other questions you wanted to surface, Dan or Hamel Husain? I'm still reading through them. I should check the Discord as well, actually.

Sorry, I was just paying attention to what you're saying. I wasn't even looking at questions.

Cool.

Okay.

Yeah. So the person who was asking about the template, they said, if I switched to 4-bit with the 4090s, I hit out of memory, but it still works if I have a sequence length. Yeah. So that, like, that's a reason why then I would think of maybe looking at CPU offloading. It's like, oh, especially now that people are really excited about longer context stuff, being only able to fit a sequence length of 200 or 500, or, you know, it's quite limiting quite quickly. And so then saying, look, let me just keep all the weights on the CPU, and then I can at least fit a nice big batch with long sequences on the GPU, and I'll just eat up the transfer time is longer, but at least I can do a lot of computation once I, once I get the weights there. Yeah, so that might be the time when CPU offloading starts to become attractive.

Um, um, I'm really curious, like, um, what kind of things do you pay attention to in terms of like reading or learning? Um, is it just according to your interest? Do you try to— yeah, like, do you kind of like rely on your colleagues at Answer AI to fill— to bubble up certain things? Do you have like—

if you—

yeah, just curious, like Yeah, I mean, I have input streams that I've quite carefully curated, right? So I have fellow colleagues who are sharing papers in our Discord or our whatever internal meetings. I have a Twitter where I only use the following tab. These are people that I follow that I get some value from, although I mean, I also have lots of other random stuff in there. I have a high-value list instead that I can go look at. And then in terms of like what from there that I'm choosing to dig deeper into, it's usually informed by some curiosity, like, oh, I'll have some particular topic that I'm interested in at the moment, either I want to teach it or I'm using it. The main driver for that that I find is helpful is having some project. So if you want to learn about fine-tuning, and then you're not actually doing any fine-tuning, I find that very hard. And so when I was first getting back into language models, I mean, after being aware of them and reading about them, all of the stuff I've been keeping up. For a while, but I hadn't been like a practitioner. And so I just went and found with Jeremy, like what competitions are ongoing. There was like a science question challenge. And that's enough to have a project where you say, okay, now I'm not just reading this paper for the sake of reading this paper. I'm reading this paper and I'm thinking, huh, I should try this on the Kaggle science competition and see if I can use it to make my training faster or use it to get a better score, you know. And so having like a pet project where whatever new thing comes out, I can quickly try it. What that project is for me has changed. For a while we were doing the efficient stuff. And so then a new model would come out. Hey, can I train this with my QLoRA thing? Is there any surprises there? At the moment I'm kind of like between projects. I'm a little bit like drifting because I don't have a reason to fine-tune language models right now. I've got some things on the back burner, but I don't have, I'm not like doing it daily. But yeah, so that'd be a good piece of advice would be like find some project that you're doing and then it's much easier to read papers because, oh, this one is about you know, multi-resolution image inputs for multimodal. I don't care. I'm fine-tuning, you know, data extraction from receipts. This one is about extending your context length. I do care because I've been running out of memory and I want longer context, you know. And so then that helps you like narrow in on the bit that's useful for the current project.

And you mentioned that, or I, and I also know that, you know, you've been a teacher and like have, you know, done courses on lots of things, including like diffusion, a lot of diffusion model stuff, a lot of like image, like, you know, text-to-image stuff and other things. Do you think you might teach again at some point? Is it open? Are you— have you closed that chapter or—

No, I love teaching. I love teaching. I feel a little bit bad that pretty much every week I'm presenting papers to some fellow researchers, but it's not in a public forum, right? It's in a private Discord chat. And then I'm also chatting with my colleagues now. And so my public output of teaching is kind of like dried up a little bit. I keep wanting to put more on the YouTube channel and things. So I'm sort of actively thinking like, how do I keep doing that? We are hoping to publish a lot at Ansa. You'll see more and more like casual blogs from me and others. as we get into the swing of that. We've also started recording like dev chats where it's literally like, this is what we're working on, small nitty-gritty tiny projects, whatever, mostly for each other, but we'll make those public. Um, so that'll help. Um, I'd definitely be open to teaching more at some point. I find it a very useful way to learn and to keep skills and to, to do research even is to be teaching something and having people asking questions. Can't we do that? Why does no one do that? You know, and suddenly like maybe today I had an idea to I should do a good paged CUDA memory thing, you know, maybe that would actually be valuable. I'm not going to do it, by the way, don't ask me for that. But that kind of idea, yeah, I do like it. Cool.

I have maybe two questions. One is probably of more interest to everyone else, and one is more interest to me. I'll start with the selfish one, more interest to me. You're probably familiar with this, like Sarah Hooker, hypothesis of the hardware lottery. Maybe deep learning isn't like the platonic best algorithm, but is instead like— we have happened to work really well with GPUs, which were being developed at the time, and as a result, um, deep learning became very popular and these things can now like co-evolve. Um, I think you think a lot more about this interface between how the compute happens and what the algorithm is than most people. Do you think that if her theory is right, or if this hardware lottery explanation like has some truth to it, that we won't have any future like major shifts like totally away from deep learning because the amount of investment into hardware that's optimized for deep learning is so great, and as a result like we will forever be, or almost forever be, like pursuing small variations off this path?

I see that. I do think there's hope and I'll explain why. It is a bit of a problem that you sort of get locked into the default path, right? Everyone's designing around matrix multiplies for attention, right? And for transformers. But the reasons I'm hopeful, one is that's okay as long as you can try the other stuff, even if it's a little slower. And so a GPU is still, you can still do it, make it, make it do a lot of things. Now it's, it's very difficult to program GPUs at the moment, and I'm hoping that changes. I think that would be a big way to break out of the, we're stuck in the default layers that PyTorch gives us, you know, if you could, if you could really teach a lot more people how to robustly program GPUs in an efficient way. I've seen amazing papers that say, look, theoretical speedup from our approach is incredible, but we can't write CUDA kernels. And that's valid. You're researchers, writing CUDA kernels is very hard. And so we only see, are things actually slower than attention or whatever? And sometimes that gets a little bit ridiculed, and definitely it doesn't make it easy to accept. And you need the rare geniuses like TreeDAO who can say, here's a new algorithm and here's the hardware-aware, efficient, fast kernel implementation of it. You know, and you need both of those things for people to say, oh, I'll give it a try and actually see adoption and speed up. Um, yeah. So I'm hopeful that maybe we'll get like better ways of programming the GPU, better ways of making that portable, something like Mojo, you know, like just enable, maybe the devices are still more optimized for the things we currently do, but like they're also getting generally faster every year. I could probably do things on the GPU now that you wouldn't think of as GPU workloads, like CPU-ish stuff, and still be a lot faster than CPUs 20 years ago, you know. But it's tricky now because you need to be a GPU magician. But yeah, so what I'm saying is like, it might lag, but I think there'll always be that off-ramp of like, oh, try some other stuff, especially if we make it easier to program. And yeah, so I think there's a little bit of hope to to make it accessible to keep trying these alternates. It's just very tough to compete with the bandwagon.

Yeah. The question I suspect, you know, like the people who join this are those who joined like this conference are broadly people who actively take ownership of improving and learning and like are trying to improve their skills. Um, is there any thread of the research that you've read recently that makes, or any technology where if you were thinking like, I want to skate to where the puck will be, that probably doesn't actually work in all geographies, but like, if I want to, if I want to foresee like and build skills for what's going to be most exciting 6 months or a year from now, like that you would jump on those, um, threads of the research or those topics? Oof.

Um, I'm not sure.

Do you even think like that though? Like, like in the sense, like, I feel like you in the, at least my conversation with you, and I feel like I try to do this too, is like you try to like find some interesting project and then kind of try to make something? Or do you think about it the other way?

It depends on what your objective is, right? If you want to do novel research, there are things that I would do that I don't think are getting much attention at all now.

I see.

But I also don't know how those turn into anything useful in the near term, right? It's really, really fun to play with cellular automata and like that kind of thing, you know? And I'm so glad that Sakana AI Labs are kind of playing with evolution and swarms and all that kind of stuff. I think that's super cool. But if I want to build a useful app today, I don't think the way to go is to start coding up some neural cellular automata. That's the fun research direction that there's not much work in. If you want to, yeah, I think the project-oriented, practical way is a very different set of things. And then you're asking, what's possible now that wasn't possible before? What's hard now that I couldn't do at all before? Hopefully it'll keep getting easier as the models get better, but at least I can start prototyping. I can put it out and see if it's actually useful in the world. I can get feedback on it. And I can learn those kinds of lessons that are very practical lessons, very like encountering the real world lessons that let you actually build a good product. And then for that, I don't think you need like some secret intuition of like, oh, actually, listen, Mamba, but with this modification, it's going to be where it's all at. Like that's research stuff. Let the researchers do the thing. And then yeah, like if you are wanting to be that researcher finding novel stuff, just staying away from the most attractive fruit. You know, like it's very easy to write a LoRA variant and very easy to test that, very easy to see a small improvement because you didn't really tune the baseline. So there's about a million and one LoRA papers that are small modifications. And some of those are really good, but it's also very crowded space. So if you can find something that's, it's a tricky challenge, but something that's doable without too much compute, but also like not the thing that every grad student is chasing to get a paper accepted on LLMs, you know, That's the— yeah, that's the harder ask.

I do think there's something that's somewhere in between there, which is to say, um, I'm not going to do something for research. Maybe I want to make a job switch in 6 months, and I expect that job switch to be as an applied— as an applied LLM practitioner. And If someone said, I actually think that working with OpenWeights models is going to be super important, then for someone who's actually sort of a beginner now and isn't going to do research, but for them to learn to use the Hugging Face toolchain would be particularly valuable. And on the flip side, if you said, I think that the main foundation models from, um, Anthropic and Google and and X and, uh, OpenAI are going to be really important, and the way people are going to use them is whatever, agents. Like, those are all mainstream tools. But if you're picking a project, like, what's the projects I do today so that my skills will be in demand in a few months? I feel like that's a little bit different than— yeah, than what's the research topic to do something truly novel.

I see. So, um, yeah, I suppose that's It's interesting. I mean, because everything's changing, like, oh cool, you spend a few months now, right? And you try and work on like context extension, right? And then in the meantime, Google come out with their 10 million context model and they open source it. I don't think that's wasted time necessarily. Like you're chasing a need that you have now. You know, you're showing that you like get some technical understanding, you're learning how to like dig in, you're figuring out what these things mean, you're writing code. Like, I don't think that would harm you in a job search down the road. It definitely does feel like there's a lot of things that you can do that do feel redundant when the next model comes out and does it well. You know, I saw this as like the image stuff. You could improve Stable Diffusion version 1.5, and I did, with all sorts of tricks, right? And then, you know, like, DALL·E 3 just always does better because it's a bigger model and it's, it's pretty cheap and whatever else. But I also don't think I wasted that time doing that diffusion model research. I learned a lot about digging into the internals of models, a lot about inspecting data, building evals, all the things that will actually transfer whether you're calling an API or whatnot. It's still very good skills to have those empirical methods and tenacity and all that good stuff.

Yep.

Oh, fun question about, like, LLMs actually impacting research. Yeah, I've been trying to use them. I mean, I've been trying to build some tools, you know, scraping arXiv and then looking for relevant papers and that kind of thing. I still, it's really hard to beat the baseline, which is when I say, oh, what was that paper again? That like showed you need to attend to the first couple of tokens. And I'm in a call with like Tanishq and Benjamin Warner and Jeremy and you're like, it's really hard to beat the baseline of like my friends can remember what that paper is much better than any of the retrieval options I've tried. But it is fun to start seeing there's a couple of like good search things now. Undermind I quite liked. You type in like a description of like what you're looking for in a lot of detail and it really does thoroughly go through and then it uses like multiple steps to say, okay, retrieves a bunch of candidate papers, has them read by an actual model that then scores whether it is actually relevant or whether it's somewhat relevant, or, you know, and presents the results to you. I found that quite useful for finding papers that I'd missed in a given field. I also like just using the, like, Chat with PDF thing for a paper I've already read, but maybe it was like a month or two ago, to quickly be like, what data did they use again? Or did they try this? You know, like that kind of quick search within the paper. I found that kind of helpful. But we definitely like, it's one of the directions we're interested in. The answer is, can we make this research a lot better versus just like, oh, the summary was kind of helpful. Like, can we actually do much more interesting things with clustering and searching and summarizing and linking and exploring research? Someone asks about note-taking, MO. I mean, I'm happy to go on as long as we want, by the way. Like, I don't know if this is— have we got 8 minutes left or can we carry on?

We won't get kicked off Zoom.

Okay.

Or— yeah. Right. Computer scientists are always like, oh, at some point Zoom will run out of RAM or something, or we'll run out of hard disk space to store a recording. So I won't say forever, but we won't get kicked out of Zoom anytime soon.

Okay, cool. Yeah, I'm happy to keep on going. I'm trying to think if there's, I don't think there's too many loose ends from the previous talk other than some things that I mentioned on the Discord chat and that. But if anyone still has like, oh, you said this word gradient checkpointing and never really defined it, can we go into that? But I am also enjoying this more like general AMA stuff. So Yeah, if you have both kinds of questions, keep on coming. Okay, let me quickly rapid-fire some. Do I still do research on GANs, diffusion models, etc., or is it just LLMs? Yeah, I still enjoy reading and keeping an eye on the diffusion space and tinkering occasionally, but I don't, like, I haven't trained a diffusion model in months. So it's more like an excited spectator. Same with, I got really into like 3D and 3D generation and Gaussian splatting and all of that. Excited sideliner, but I just don't have the time. Also excited about lots of non-LLM stuff in terms of building out an R&D organization. So that's quite cool. What was the search tool you just mentioned? Undermind. So they're, I think the free version is pretty good. What was the AI lab that does evolutionary stuff? So this is a guy called David Ha. And his lab is called Sakana. S-A-K-A-N-A. It's a Japanese, Japanese research lab. Maybe I've got that wrong. Yes, Sakana. Sakana.ai. Um, they did this paper on evolutionary model merging. It was a really nice paper. It was one of their first research outputs. But I know them. They're interested in all sorts of interesting, like, collective intelligence stuff. Um, a bit off the beaten track versus the standard foundation model company.

There are a couple of people who just casually commented that they'd like to watch you, a stream of you coding. If they did that, is there anything that they would be struck by or surprised by about your workflow when you code or your style?

I don't think so. I think, like, I'm not an amazing coder. I take a little while to get things. I rely on Copilot to write a lot of my boilerplate. I could try streaming sometime, but honestly, it's going to be a lot of like, okay, I'm reading the documentation for this thing and I'm typing this thing. I'm finding my coding style is shifting a fair bit at the moment because I'm, I've been self-taught solo for a very long time. And now I'm working with some amazing people and just a lot of like, I don't know, I never knew there was a keyboard shortcut to do, and then it'll be like some super obvious thing. Cut the whole line. You don't have to select it first. You can just say Ctrl+X. Like, whoa, that, you know, okay, cool.

I don't know.

There's lots of little things that you'd probably be frustrated if you already knew them. Like, oh, why is he stenotyping that whole thing? Doesn't he know there's this shorter function? I've been watching in awe how fast my colleague Jeremy Howard can build stuff and trying to learn from him, like techniques for iterating and exploring stuff in notebook and building stuff up quickly. But yeah, I don't think it would be particularly interesting.

Are you using nbdev yet to build something? Or do you still prefer VS Code or any preferences? Or I don't know, the question could be wrong, by the way.

I don't want to— Yeah, when we're building stuff that's it's going to be packaged up. nbdev is really nice. So yeah, I guess that's a thing people could check out. Maybe that is quite different to normal. I do sometimes write things still in, like, as a script. It really depends on the application. I will, if it's a web server that needs to be taking requests or something like that, I'll just write in a script and Jeremy will be like, how can I completely change how notebooks work to make this work in my notebook? And those are both really fun. Yeah. So, okay. I do a lot of coding in notebooks. Almost all my exploratory coding is in a VS Code Cursor Notebook, although I have been trying to also try out Jupyter Classic for— because we have some extensions that run there. Yeah, I use AI Code Complete a lot. I try and comment. I write like I'm writing a tutorial. It's from a lot of teaching, and that really helps with Copilot and things, you know. If you're very terse and compressed, the Copilot sometimes has a harder time predicting. If you're writing like you're writing a tutorial, which is my default, It's very easy for it to predict what I want to do next often, so that's quite nice. But yeah, I don't think there's anything special there. Just looks like a somewhat beginner Python coder banging his head on things.

When you say terse, are you also talking about the code, not just the comments?

Yeah, yeah. Sometimes you can do lots of things at once with some very clever list comprehension. Okay. And that's really nice, but I sometimes like to be more explicit.

Jeremy's code is extremely compact.

Yeah, yeah.

Yeah.

And what you got to realize as well is like you see the final result of people's coding and even Jeremy writes things out first and then compresses them down. So yeah, the only difference is that sometimes, especially if I'm going to be teaching beginners or if I'm wanting to coach Copilot, I'll leave it nice and explicit, even if it's 5 extra lines.

Yeah.

Cool. Let's see. Oh, do you know if QLoRA or bits_and_bytes introduces some quantization overhead specifically for long context? You know why? Yeah, I mean, so this is one of those places where the napkin math says one thing and the actual result something different. All of the quantization libraries have some computational overhead. So the total amount of number crunching I have to do goes up. Which is why, for example, on an H100, LoRA might be faster than QLoRA. Even though you have less weights to copy around, you have to decompress those weights, and actually the bottleneck is the number crunching. In addition, with QLoRA, especially with reentrant_checkpointing=False, I want to say, there was, for some reason, the activations were just larger than they would be in the LoRA case. And so we had, especially initially, like, oh, QLoRA is better than LoRA at low context length, but as the context length grows, the LoRA is actually like the more memory efficient one and the QLoRA takes more memory. I think they've fixed that or at least tracked it down to being related to the gradient checkpointing implementation. I'm not sure what the status is, whether that's fully fixed or whether it's still something you might step on. Yeah, so there's always things like that that are unexpected, right? Like some bug, something keeps, is in the wrong precision or something gets duplicated. So you do your calculations and you get one answer and then you see in practice it's slightly different. Yeah, so that was one that bit us was the gradient checkpointing, whether or not it's using reentrant checkpointing with QLoRA was throwing us off on long context. But we managed to get it fixed. Cool, nice comments about teaching. Question on note-taking. I am a mix. I've got a physical notebook with me all the time. I use Notion to jot some things down. We keep our papers in Zotero and we can add notes there. Yeah, I write a lot of things that I eventually put up as blog posts and others that just stick around as drafts. We've got some internal blog posts that answer, so it tends to get a little scattered, but as long as you're writing stuff down somewhere that you can find it again. Cool. Should we keep on going? I still see a trickle of questions, but don't want to hold everyone up the whole day.

Yeah, we got one, we got one, someone asked both in Zoom and Discord, so we should answer this one. What are your thoughts on 1.5 8-bit LLMs?

Oh, nice. I like it. Okay, so to give everyone context on this, we'd said, oh, you can compress the weights of the model by quantizing them. And we said, oh, you could do 8-bit or even 4-bit with some tricks. As you're compressing those weights, you're losing information. And often you find that then you get a performance degradation as a result. And so even 4 bits seemed kind of crazy. And to do it, you have to keep track of like how I'm scaling things. Maybe you store some extra numbers, which is why it's like 4.2 bits per parameter, right? Because every parameter gets 4 bits, but then I'm also storing like a scalar for that group and an offset and whatever else. So Microsoft had a paper called The Error of 1-Bit LLMs where they had every weight became either a 0 or a 1, but they were able to then still get some sort of performance out of that. It just wasn't as good. And then they said, oh, fine, instead of 0 or 1, we're going to really splurge and we're going to use either -1, 0, or 1. So that's where the 1.68-bit per parameter thing comes from. And that's, that's kind of funny because you're compressing down all these weights to basically 1 of 3 values. Where I'm excited, just that paper and some other things, it feels like that's not enough if that's all you have. Like, you take your model, you quantize it down, I think you're going to get atrocious performance. And I don't know how you get around that because we're training these models longer and longer. They're using the nuance in those things more and more. But if you have most of your weights down in 1.68 or 2 or 4 bits, 3 bits, whatever. If you, if you compress most of your weights down, the model does worse. But then if you can add a few parameters back, and maybe they don't have to be high rank, so maybe you can use something like LoRA or DoRA or whatever, I think you can account for a lot of that loss. And I think there might be some trade-off there that's beneficial. So like maybe, look, I want the highest accuracy possible, fine, I'll keep it in 16-bit. But then I'm using up a lot of memory. Okay, cool. I can quantize that down to like 4-bit and it's still kind of okay. Maybe there's a place where we say, look, we have -1, 0, 1 for the base weights. We have a LoRA adapter on top that kind of corrects for that. And we have really efficient kernels that can now take advantage of the fact that those -1, 0, and 1 operations are a lot easier than multiplying arbitrary floating points. Maybe we can do some really hardware, you know, very efficient hardware stuff to do those operations. Then the higher precision adapters we can do separately, and that can also be quite fast because there's fewer parameters. And those two together in some sort of fused kernel, I think, could be really fast, decently accurate, and a lot less memory. So I think that's like a nice trade-off. Whether anyone's nailed it yet, I don't know. Like, I think maybe we'll find training techniques that make it even more amenable to that. Maybe it turns out that the trade-off is like, no, actually, that's too aggressive. You'll always do better with a smaller model in 4-bit versus a larger model in 2-bit. But it seems like there's something there. Even if it's like, oh, some of the model is in 1 or 2-bit, the important attention layers are in 4-bit, you know, the layer norms are in 16-bit. Like there's some juggle there that's really good. Yeah, check out the Moebius Labs they have a post on their work. They've done fantastic stuff. I think they've basically shown some proof of concept for this. Someone's just linked their HQ version of Llama, but like check out their like blog posts and things as well. They're doing really good work and we're doing some collaborations with some of them, I think, to try and get those fast kernels. Okay. Cool. Some questions about some of these alternative architectures. Can SSM, XLSTM, Markov, I guess there's different things. One is these different foundational building blocks like the KAN, or you have these rather than a big matmul, maybe I'll have a tree of things. Those are very exciting. I feel like they're still kind of researchy stage, and I don't know that many of them will pan out into actual practical usefulness. State-space models and these recurrent models, and we were chatting about this, Dan, offline, those I think are very cool. There's definitely some types of operations that attention is more suited to that maybe they'll struggle, but also if they scale well enough and fast enough, I think they'll be able to compensate. We're already seeing fantastic text-to-speech just came out based on state-space models. I know someone working with DNA, ultra-long sequences, Yeah, it's really cool to have that other tool in the toolchain. I don't know that they'll always surpass transformers or anything like that, but they do seem exciting and a nice little research niche. But also, I don't think it changes your dynamics much as a practitioner. Like, oh cool, I replace LLaMA with Mamba and I get the same or different performance on my benchmark. That's cool. And it's just really neat that they enable some longer context stuff. Right, well, we could go on, but maybe we should start wrapping up. Yeah. Thanks everybody for the questions. That was really good to get. I think I felt like I rushed a little bit on the first one, so it was nice to have some leisure to just make sure there wasn't loose ends and go into all the other stuff. That Napkin Math channel on Discord, I'll keep an eye on that. Maybe like, I mean, I'll be away in the weekend, but I'll try and keep an eye out for extra questions there, especially as people watch the recording. But yeah, anything else, Dan?

Nothing else. This was great. Thanks so much.

Yeah, my pleasure.

Yeah, see ya.