Deploying Fine-Tuned Models

We're warming up. I think today is one of the— at a technical level, one of the deepest topics and also most complex. So thrilled to have this group here, and I think a bunch of you are going to— I think everyone's going to learn a ton today, so I'm certainly looking forward to it. Our plan is I'm going to give an overview of serving and talk about some of my experience. I think that because of the usage patterns of problems that I've worked on as a practitioner, it had— I've been able to avoid some of the complexities of real-time serving. So I would touch on those things, but we'll get into a lot more of that as we go further. Hamel Husain worked on more real-time use cases, and then we're going to go through Travis. I think actually this is out of order.

Yeah, we're going to do Joe, then Travis, then Charles.

Yep. So that's our plan for today. And with that, we will get started on talking about how to serve and deploy fine-tunes. There are As a first approximation, two types of fine-tunes, but we have focused on one of them so far. So there's full fine-tunes where you potentially are changing all of the weights that were trained in the original model. I'm going to set that aside for the moment and instead focus on deploying or serving models that were fine-tuned with LoRA or low-rank adapters. And as a recap, so this is a slide from Workshop 2. If you look at any given layer in your model, this is even simplified a little bit from that. So we're not thinking about the separate query matrices. But if you imagine for any given matrix that goes from the input in a layer to the output in the layer, if the input representation was 4,000 dimensions and the output was 4,000 dimensions, then a matrix that goes from one to the other would be 4,000 by 4,000. That is 16 million weights, 4,000 times 4,000. If instead you are training with one of these low-rank adapters, then you'll have one matrix that is 4,000 by some lower number. So a potential adapter rank would be 16. So it'd be, this lower one might be 4,000 by 16. The next one would be 16 by 4,000. If you 2 of those, then you can do some arithmetic here. You see, you go from 16 million weights that you would tune if you tuned all of the original weights, but instead in this adapter, which has a particular type of bottleneck or restriction, that is 128,000 weights. And most of when we talk about fine-tuning, we're actually typically talking about these LoRA fine-tunes. And after you have trained, you need to make a decision. One thing you can do is say, I'm just going to keep that LoRA, those weights in a separate file. And then I'm going to deploy with that. That would be sort of keeping it in the before stage. The other thing that you can do is you can do that multiplication of the, these two weight matrices, add them to the original weights. And merge, create new, one new file which for each layer has the base model weights plus the fine-tune, find the learned weights during fine-tuning. That really takes us to a few different potential workflows. So you say I'm going to fine-tune within that, I can fine-tune with a full fine-tune, or I can fine-tune with LoRA, and then with LoRA I can either do that merge that I just described, or I can do something which others in this group have immense knowledge about, um, which is hot swapping the adapters. And I have only the most superficial knowledge about that, but I'm going to let them talk through a lot of the details of that. But as a practitioner, the things I think you should know are fine-tuning if you have enough data, like, it does have some advantages, but it is memory intensive at training time, and so it is not so, so widely used. Merging the weights of the base model to your LoRA, so it's all in one file, it's all in one either file, or if it's very large, one sharded file, that is historically quite widely used. Um, my— that's what I've historically done as a practitioner. Um, and then, um, the arguably the new hotness is this hot swapping adapters, and that has some potential efficiencies, especially for someone who is serving many different models, or rather many different adapters.

And the, the one that's widely used—

why the reason it's widely used I can't tell if I have a bad connection or if Hamel does. Okay. Even though I think Hamel's trying to talk, I'm just going to talk over him since we can't really understand him. Um, he was going to say that it's why they use it. I think that's primarily for two reasons. One is a lot of times people are serving a small number of models, and as you get to platforms, they can— I'll talk more about what platforms can do in a moment. And then the other is that historically, um, this hot swapping of adapters, uh, yeah, it's a newer technology, and I think that some of the economic advantages of it, which I'll talk about in the next slide. They just haven't been realized yet. So, um, when you are deploying a model, really the biggest trade-off that you have is performance versus costs. Some of these are not worth going into much depth about. So for instance, if you have more powerful GPU, and obviously you can have a lower, um, lower latency. We talked about either in Jono's, um, Charles's office hours that a more powerful or faster GPU can just get through the work. So just looking at the cost per hour is not the right comparison. If you were using these things at 100% efficiency or 100% capacity, then using a more powerful GPU might be more of a pure win, but there's just waiting time between requests. So having a more powerful GPU is more expensive Running a larger model, it just is potentially higher performance in terms of quality, but then it's slower or you need more GPUs to run it quickly at the same cost, more powerful GPUs. There's some engineering wins which really are mostly at the platform level, so I'm gonna set those aside. And I wanna talk mostly about this last piece, which I think is like the fundam— If I had to pick one tension, that we need to think through as practitioners, I would say it's this cold start versus idle time trade-off. So loading a model onto a GPU takes— there's just so many, so many different even definitions of what we could mean by loading a model onto a GPU. But I think And like, it can take anywhere, depending on the details, from 30 seconds to, I don't know, 8 minutes. And so it's, if you have like a real-time use case, these cold starts are just a showstopper. The flip side of that is that if you go a minute, if you, if you have a request and if you have the model loaded and that would take let's say a second, but then you will wait, you wait a minute before your next request comes in, then you're paying for 60 times as much GPU time as you actually need. And for newer products, you could actually have a worse ratio than that. And so this idle time is particularly costly, and it could mean that you're paying for a somewhat expensive GPU or group of GPUs for, you know, 50 times as much as you actually need. And so you need to decide, like, do I need low latency and I'm willing to pay for that idle time, um, or is there some other solution? And when others talk about hot swapping, um, this is basically some— a way of trying to achieve economies of scale of having many LoRAs, um, being served off the same GPU so that you have a more constant, um, traffic to that GPU and you can overcome this cold start versus idle time problem or challenge. Okay, I want to emphasize that especially if your work is with technology companies or if you're just used to thinking about something like ChatGPT as the canonical use case or Copilot, you probably assume that most use cases are real-time. Um, the, I think that the projects that I, the commercial projects that I've worked on are probably not really representative. And there's some, some reasons about my clients that are, that are the explanation for that. Um, but my projects are mostly not real-time. So I've talked about several of these before. So there's writing alt text descriptions. We have a queue of images that comes in every day. Every morning we start a job, we process those as a batch, and, and I'll show you like some of the setup for how we do this, just the code and such. But yeah, we process those as a batch and then humans review all those during the day. But once we've processed that batch, we scale the GPU down to zero and then we don't have any more requests until the next morning. Extracting chemical properties from academic chemistry articles that then— and that all goes into a structured database. Again, we just do that in batch once every morning. I've talked about this project I worked on editing journal articles to identify and potentially and propose edits in stereotypes or unconscious biases. Again, those start as a queue of documents. Each day we run that queue and then people review the edits over the course of the day. The one project that I worked on that was real time was this super popular or fashionable use case of allowing employees to ask a question in plain text, and then you convert that into a SQL, run the SQL, and then give an answer to them. The project that I did that on, we used OpenAI, and so we didn't have to build or even think deeply about, um, how that, how that large language model got served. But the ones— these top 3 are examples of projects that I worked on where we used open weights models, and for all those we just, um, batched everything together and did it once a day, and that made our life, um, much simpler.

In the, in the Texas SQL, that's really interesting, um, like with the Honeycomb example It's kind of like, yeah, it's basically text-to-SQL, but we used OpenModels, and so we'll be talking about that.

Yep.

Um, I'm gonna go through the, uh, somewhat briefly through the workflow that I personally have done for those top 3 use cases. It's pretty similar across all of them. Um, and those are— so here I'm showing a screenshot at the top of what does my my LoRA directory look like after I finish training. You'll see that I've got an adapter model binary. It's got some weights in it, but 168 megabytes of weights, which is, compared to many things we'll see, not so great. I have a step where I merge that to the base model, and that creates the directories that you see at the bottom. You'll see that we have 4 PyTorch model binaries. There's 4 of them 'cause they're sharded 'cause they're big files and that's 16 gigabytes of weights. So we have something like an 80x increase in the weights files and as a result, while I could load 168 megabytes onto a GPU quickly and maybe already have the base model stored on that GPU, which is this hot swapping idea that others will talk about, Loading 16 gigabytes is a lot. So after I have done this step, which is just a one-liner, and you can even see that one-liner in the future, and this one-liner is in the README on the Axolotl GitHub page. So like so many things, you can look there in the future or you'll have access to these slides, but I would look there to see the command. From there, we've historically serve things with, um, Hugging Face inference endpoints. There's nothing magical about, um, Hugging Face inference endpoints. Uh, 3 other people on this call have built platforms that are also quite good. This is just the one that, uh, we started with and haven't moved off of, and I think is also quite nice. And like many others, you have credits for. So the steps here are to, uh, create a repository. The rest of this stuff is like basic Git commands. So I created a Hugging Face repository called conference_demo, copied the merged files into that directory. This git lfs, for people who aren't familiar with it, that's just a Git large file alternative workflow for handling large files. So this is just saying to track all any binary files because those are about 4 gigabytes each. I add them, I commit them, I push them to the Hugging Face repository. And then from there, I'm going to show you in a video the next step. Again, there's so much complexity underneath everything we do, but you're going to find that this is reasonably straightforward. I'm gonna even show it to you in a GUI. So let me start that. I just wanna pause and see, yeah, I can't get too much high resolution, but I've got this repository. From there, I hit the deploy button. You'll see there are a handful of options. The one that we've historically used is inference endpoints, which is a dedicated endpoint. So you'll see I will select that. From there you can choose what cloud you're on. That's frequently— you want to be on the same cloud that your company does most their work on for a variety of reasons. You can choose what GPU you use to serve it. And then I think the last thing I want to spend a little bit more time on is Here we're choosing automatically scale. You can say I want to scale to zero never. That would be— this is just always up in that trade-off between accepting cold boots or cold starts and paying extra for idle time. Here you just say like we'll pay for the idle time, we'll always— we'll never have cold boots. The thing that we've done because we're doing everything in a batch. I'll select to scale to zero after 15 minutes of inactivity. And now we probably only pay for the GPU for an hour or so a day. That is the hour or so when it's actually getting used after we've finished with our batch of requests. And those are just simple POST requests to the endpoint they provide. Then, You could manually shut it down, but we aren't even that careful about it. We just let it scale to zero after 15 minutes. So, uh, yeah, this has historically been our serving workflow. There are, uh, things that are much more sophisticated, especially as you get to real time. We'll talk about those, but I think of this as one of several reasonably simple ways to just get your model served, um, and it's been sufficient for most of what I've done. Uh, so I'm going to at this point hand it off to— in some ways we're actually like increasing complexity as we go. So I'm going to talk— hand it to Hamel, who will talk about one level deeper in complexity, and then we'll hand it to, uh, we're going to move to Joe and Travis and Charles, and they'll all be a level of complexity beyond Hamel.

You have to make me a co-host again because I love Tinkerbots.

Cool.

So I will make you a co-host and I'm going to stop sharing. Uh, one other comment I'll make, um, is that we're going to— I think our order or plan is, um, we're going to do me, Hamel Husain, and I think Joe, and then after that we're going to do a round of Q&A and then we'll do Charles and Travis, and then we'll do another round of Q&A.

Um, I'm gonna talk— okay, thanks, thanks, Dan. I'm gonna talk a little bit about model deployment, uh, more generally, and I'm also show you some code. I'm gonna go into the Honeycomb example and how I actually deployed that in real life. Um, so first, um, I'm gonna talk a little bit about model deployment. Um, actually, let me turn on my camera. Sorry about that. Okay, great. So, um, there's a lot of things to think about when you think about model deployment, and there's a lot of different dimensions, um, in which things can get more complex. So here's a quick chart. You don't need to memorize this. This is just to give you an idea. So things like speed, you know, is it okay if results are slow or do they need to be fast? Scale— do you need to handle lots of requests concurrently or do you not have that many requests at the same time? Another thing is pace of improvement. Do you need to constantly fiddle with your model, constantly update your model, things like that? Do you have real-time inputs? You're constantly streaming inputs through your system. Also, reliability is an issue. Like, you know, if— is it— does it have to be high availability? Is it going to be catastrophic if your model server goes down? For some period of time. And then like the complexity of your model, like, you know, what are your resource requirements, how big are your models, things like that. And so kind of the left-hand column is where you have like the very simple use cases where things don't get too complicated. There's a lot of off-the-shelf tools. The more things on the right-hand side of the column that you kind of stack, then the more complicated it gets. And if you kind of stack too many of those things, then sometimes you need to build custom, custom stuff. The tools are always getting better. So you might not need to build custom stuff, but it's just these are things you want to keep in mind when you like look at your use case. So like, what am I talking about with like simple versus complex? Let me give you an idea. So like simple model serving is something like you have like some application, maybe some website. And you just have like interfaces directly with your model library almost. So it's like you can just imagine putting your model in FastAPI, something very simple. It's not doing that much, you know, simple model serving. It's not going to necessarily— you're not caring about high availability or speed or something like that. Potentially you just want something simple. It's really good for proof of concepts and things like that. Advanced model serving kind of can look like something like this, where you have some application, you have like some kind of auto-scaling cluster that sort of has many different model servers, and that has like load balancers that then, you know, route requests to a model server for like for high availability and kind of like, you know, specialized components for preprocessing and post-processing, things like that. Um, so like one example of this is like, you know, if you want to use Kubernetes with like a Triton, NVIDIA Triton front end, um, and, you know, maybe a TensorRTLM backend, you know, that's like one example of like a kind of architecture that people use at scale. Um, we're not going to go into this scale thing too much. Um, we'll tell you about it, like You know, Joe especially has worked a lot on building these systems at scale, and he can talk about this. But just to kind of give you like a sort of flowchart of how to decide. So like, there's many different kinds of model serving, and don't take this chart too seriously in terms of where I put the logos. But basically the idea is like, okay, is there— first of all, is there a finite— if you have a fixed set of inputs that are known in advance, which is not usually not the case with large language models, you can pre-compute your responses, or you can at least pre-compute some set of the responses and not do inference at all. But that usually is not what we're talking about, especially in this course. And then the other decision criteria is like, is it okay to return responses asynchronously, like maybe in minutes? If so, you can do more of this like batch response here on the left-hand side. If no, you need to do some kind of real-time response. And the question there becomes, are you comfortable operating these inference services yourself? And if, and if you're not, or yeah, if you're not comfortable, then use some kind of hosted service like, you know, Anyscale, Fireworks, SageMaker. But if you are comfortable with doing that, then the question becomes like, do you require large scale and low latency? You know, if you don't require that, you might use a simple like stack like FastAPI. If not, you might want to use, you know, if you do require low latency and some kind of scale, you might, you know, go with something like the NVIDIA stack, which Joe is going to talk more about. Now in this course, like, we've highlighted, you know, so I've deployed this Honeycomb model in Replicate in real life. So I'm going to be walking through that. And then we've also gone— we've also talked a lot about Modal for fine-tuning. Turns out you can also use Modal for inference in many different ways. So I'm not going to tell you where these things go on this chart. I don't want to shoehorn these folks into, like, one place here. I'll let you decide where you think it goes. But the idea is like, you know, it's kind of this decision tree of like what to use. Now, this is a benchmark that's like on a very GPU-poor kind of thing, just like one request, a batch size of 1, and just like a single request after warming up the GPU. Don't take this benchmark too seriously. All benchmarks are wrong. Some are useful. But basically, you have to— like, I— the point you should get across here is like you should try different inference servers to see what works best for you. Thing I can say at a general level is VLLM is, is like really easy to use and it has like good trade-offs. And I— and like out of all the inference servers out there, and I've tried like so many of them, I really like it a lot. The NVIDIA stack is more performant, like, so when you talk about TensorRT LLM backend, um, and then the Triton frontend, there's like backends, kind of like, um, you know, compile your model, um, kind of like are kind of where the computation is like happening, or most of it. And then like the frontend is where it's like, you know, you're getting the requests, you're handling— you potentially handle things like batching and things like that. Um, And so the NVIDIA stack is like very performant, but it's actually really difficult to use. There's a lot of knobs and it can be a little bit painful. And like Joe will actually talk a lot about that. And yeah, you want to pay attention to things like quantization. Quantization can make a huge difference. And, you know, you want to— when you do quantization, you want to run evals as well. So we talked about evals in the last lesson. A bit. So, um, you want to like have a reasoned way of thinking about the performance trade-offs. Like, okay, you can achieve lower latency by quantizing your model, but you just want to double-check like what the quality hit on the model is. Um, again, it's not important to pay too much attention to this. I just want to bring up the fact that you can try different tools and get very different results when it comes to like things like speed. And you can go— you can get really advanced with these benchmarks. There's all kinds of different ways you can slice and dice these benchmarks. So, uh, this is— this doesn't mean that one thing is better than the other. Um, if you want to see like all these slides, kind of the— I go into more detail about these things in this blog. Um, so you can just go here and you can read about a little bit more long form. Um, the next thing I'm going to talk about is the Honeycomb example, which is the through example I've been using in this course, like kind of the case study of like, uh, the, um, basically a natural language to query problem, uh, the Honeycomb query language. If you don't know what I'm talking about, I recommend looking at the last courses. But, um, so As kind of we hinted at, the Honeycomb example needed to be a real-time use case, and we wanted to, um, you know, yeah, basically it's real-time because users want to write questions and it's used in, uh, in the Honeycomb interface and get results right away, and it has to be available 24/7. And so the platform I launched this on is Replicate. So like, why Replicate? Let me just like talk about it just for a second. So this is Replicate. This is kind of the page where I deployed the Honeycomb model to. And I'll, I'll talk about like how the code— I'll show you the code of like how it got here. But basically there's some reasons why I chose Replicate in this example. One is a lot of times when you're working with non-business people and you want to have them try out your model, it's really nice to have like a playground. and I want to— a lot of times that playground is not just a prompt, it's like some kind of structured input, and I want to guide that structured input. So in this case, I have the natural language query, and this is the schema. So remember, there's two inputs into the language model, um, and you know, this one is usually retrieved by RAG, but for the purposes of like playing with the model, it's really useful like when you push your model to Replicate and you define it in your Python, like the predict function, like it'll create this entire user interface for you. And the thing that's nice is I can give this to my counterpart and I can have them play around. But then also like the predictions have permalinks. So like my counter, my client Philip, who I've worked with on this Honeycomb example, you know, could kind of like play around and send me situations where he thinks something was kind of weird and he gave me a permalink. And actually, like, the permalinks are like really useful because, um, I'm just going to go ahead and run this. But like, and this is what it looks like when it's run, um, you know, you get a link to this prediction. You see like the URL here. And every time I make one of these things, like, there's so many degrees of freedom. Like, you might, you know, like, for example, what if you put this in double quotes, the name of the columns? Or what if you did something funny, like put newlines where you weren't supposed to? Well, those permalinks help me debug that. Like when my, when my counterpart sends me something, I'm like, oh, you, you actually like did this wrong. Another thing that I really like is like when you, when I share something with somebody, it actually comes bundled with documentation. So you can go to the API kind of page here and then basically it has like a full sort of code they can just copy and paste, including the inputs in that example. So like, you know, you can just basically copy-paste this, this, uh, like Node.js code or this Python code, or just make a curl request for that specific example. You can also like save examples. So I've saved some examples here. So all this stuff is really nice, and that's like essentially why I chose to do it. It's actually— and it's also really easy. So that's why I ended up for this specific example choosing Replicate. So let me go back to this. Okay, so like, let me show you the code. So if you want to see how that this Honeycomb model got to Replicate, there's these two links where you might want to know about. There's this GitHub repo which you already have access to. Um, and it's actually the wrong repo. Let me go to the right repo. It's called ft-course. Inside ft-course, you'll see replicate examples, and there's these two examples. Um, I'm going to be walking through this one. That's the quantized model. It really doesn't matter, but I'm just going to show you the code right now. So let me switch my screen to VS Code so I can show you the code live here for a moment. Okay, so I'm in that folder and just want to walk you through a little bit of how Replicate works, and Joe will go into a lot more detail. Um, but, um, this is the README. Um, but basically first there's two things you need. One is this cog.yaml. So Cog is a wrapper around Docker. Um, now if you're like me You might think, why do you need a wrapper around Docker? Docker is perfectly fine. I kind of thought that too when first encountering Cog. But actually, like, it helps avoid a lot of CUDA footguns. Like, even when I'm using NVIDIA Docker or like with the Docker with like the, you know, GPU runtime, I constantly have to fight like CUDA problems. and, uh, the folks that made Cog— and Cog is this, like, open-source project that Joe will show in a little bit, um, you know, you can basically specify, uh, it basically is a Dockerfile with some other commands, but essentially what it does, it's a Dockerfile with a bundled web server in it. And so what you do is you specify predict.py So this is the predict.py function, and basically there's really not much going on here. So there's a prompt template, there's a setup. You, you, uh, define this predictor class, and then you have the setup, and the setup just basically loads the model onto the, you know, into the GPU. And this is only run upon boot or startup. And so that's this code. This is just VLM code here. So pretty straightforward. This is a, this is a model I've uploaded to Hugging Face Hub and has been quantized with AWQ. So that's where this sort of, uh, that's where these arguments come from. And then you have this predict function, and that's what's going to be called the inference time. And so you have, you know, these various inputs Um, so there's the natural language query, this is the column, and then basically this is kind of standard. This is also standard VLLM code, and I'm just emitting the output. Okay, so that's the two pieces. That's all you need: cog.yaml and predict.py. You have to go install Cog, obviously, before you do this. Um, there's a repo for that. Joe will show you a little bit more about Cog. But it's pretty straightforward. You just have to install it. And kind of like, this is how the process works. So if you want to play with this, you know, or, you know, I actually recommend that you kind of like upload your own model. Like, I don't know, the Honeycomb one may be interesting to you, but if you— a lot of people are fine-tuning their own models. I would actually recommend fine-tuning your own model or find a model that's like personally interesting to you. It'll just make it a lot more fun. So, um, if you're going to debug this, like, CoG stuff locally, um, you want to download the weights. So that's what I've done here is basically this is a command that allows you to download the weights really fast. It's this kind of long, but actually, like, it's better than cloning, like the git clone. Um, and then there's, there's different ways you can debug. Uh, one way is to run you can actually run the Cog server. So like, I'll do that here. Let me just make this window a little bit bigger. Let me clear this. And so you can actually like get an intuition for what's happening by actually running the server. Now it's cached because I already built the Docker container. I didn't want you to have to look at that for this class. And so it's going to run this web server. The web server is basically gonna, you know, it's basically the same thing that Replicate does, like what I showed you earlier, but it's just locally. And then you can send that web server request. You can see this request and you can see the query, the Honeycomb query being returned here. That's one way to run things. Another way is there's this— let me just cancel that, stop the server. There's also this cog predict function. That's one way to just, you know, if you don't want to do this server, run the server, you just want to run a, like, get a prediction, you can just pass in the inputs. So that's what this -i is here, and that's how, that's how it works. Um, and so yeah, eventually it'll come with a prediction here. You'll see it right here, same thing. There you go. And yeah, that's pretty much it. Now the way you push it to Replicate is you do this cog login. I'm not gonna do cog login. If I do cog login, you all will see my token and don't want to show you that. But it's very easy. You do cog login and then you follow that with a cog push and it pushes it to Replicate. You have to specify a destination. So actually in the Replicate, in the Replicate platform, you have to actually create a model. So you have to go to models, you have to create new model, and you kind of fill out some information. One thing you might want to— oh wait, I'm not showing— let me show the other screen. Sorry about that. Let me switch back. Okay, so like in the, in the Replicate platform, to go to, you know, on your account, you can go to models here and then create new model and you just fill out some information like a name of the model. And then the most important thing you want to select the right hardware. So depending on your model size, make sure you think about based upon what we learned in the last lesson of how to estimate memory, pick the right GPUs. And then you— yeah, I always just for this class, you can do custom cog model, at least the code that I showed here and then click create. And then that's the destination that— let me switch back to the VS Code real quick. Um, let me see if I can do it fast. Here we go. Yeah, so that's the destination you see here is kind of like this ra.im. That's just Replicate. Um, that's the Replicate repository, and then you just have the fully qualified name here. That's— it's basically like GitHub. In that sense. You have like an org and then like a name. Um, so yeah, that's how it works. It's very simple. Um, there's two different examples here. One is with the AWQ quantization VLM. Um, it's pretty straightforward. Or actually, let me, let me show you. Okay, so let me get this window out of the way. One second, the zoom is in my way all of a sudden. Okay, so, uh, yeah, let me go back to predict.py. So you see, like, I had to do a little bit more ceremony to get the quantized version in here. With the one that's not quantized, it's almost the same thing, they're just slightly less arguments. Um, and so let me switch back to the Hugging Face repo real quick just to orient you.

Um, go back to this window.

And so, um, if you want to know how it was quantized, I have this repo with the sort of— this is just like standard VLLM, um, this is how you can quantize the model. And this is what I did. I basically loaded the model, quantized it, saved it, and then uploaded it to Hugging Face and then pulled it down. Um, so that's kind of like a Lightning round of like how I deployed this Honeycomb model. Um, and then yeah, I think now is a good time to give it to Joe, who can talk a little bit more about Replicate. I kind of glazed through Replicate, uh, very fast, but he can give you more color and also tell you a lot about what he's learned, uh, especially in the NVIDIA stack and like deploying, like, uh, creating, uh, inference platforms. Let me stop sharing.

Great, thanks Hamel. Let's see, when Hamel asked me to, to talk during this, this course or this conference or this spiritual revival, I'm not sure what it is at this point, um, he gave me two 2 suggestions. One was to tell some more stories, which I have many, many are shared with Hamel. And then 2, do a demo. So I thought a lot about what, what information I could share in tens of minutes that would actually be helpful. Um, Dan Hamel alluded to the fact that serving is, is really complicated. It's really hard and sometimes it's really awful. And there's a lot that we could talk about, but I'm not sure how much it would be, how useful it would be. And so I thought a lot about if I was just starting out this journey or I was halfway through it, what would I want to know? What would, what would I wish somebody had told me? And so I'm gonna try to talk through the things that have hurt me the most, and hopefully that'll be helpful for some of you. And then I'll talk a little bit about, about Replicate, how it solves some of those problems for me, and how you can serve language models in Replicate. First share. Can you all see the slideshow? Okay, great. So we're gonna talk about deploying language models, and then we'll talk a little bit about deploying language models and Replicate. So I've been working with language models for about 4 years since the Transformers paper. Well, generative language models, been deploying them for almost as long, and things have changed a lot. But one thing that hasn't changed is that it's really hard to deploy language models. The ways that it's hard has changed, but it's still really hard, but maybe not in the ways that you think. The two ways that I found it to be quite difficult still, even in 2024, is the fact that performance is multidimensional and zero-sum. And so Dan and Hamel Husain talked about a context where You, you just— a simple deployment. You just need a model that emits tokens. You don't care too much about performance. You don't care too much about cost. If that's the world you live in, deploying language models isn't actually too hard. There are a lot of great serving frameworks. There are a lot of great platforms. These tools are ergonomic. Some of them are well documented. You can have a decent time and not experience too much pain if you don't care about performance, if cost. The problem is if you do care about those things, then suddenly everything becomes very complicated. And as I'll talk about it in a bit, performance is multidimensional. There are some dimensions of performance that you can prioritize, but if you do that, then you're penalizing other dimensions of performance. And so you really have to think carefully about what your context is. What's your use case? What does it mean to have a performance SLA? And what do you have to do to meet it? There will be trade-offs, and you probably won't discover all of them until, you know, you're shipping in 30 minutes. Then you have an emergency. And so they're multidimensional and they're zero-sum. So if you prioritize one dimension of performance, you often are penalizing another dimension. So the way I deal with this is to try to be really careful and think clearly and carefully about what it means to have a successful deployment. What's important to my users? What's important to the platform? What promises have I made to a product manager or the person who actually pays for the GPUs? The other issue is that technology never stops evolving. I've been serving language models for years, and there used to not be very many options unless you rolled your own stack. There were a couple serving frameworks. They were all hard to use, but that was it. You didn't have to think about technology change. The rate of evolution was pretty slow. That changed dramatically last year, 2 years ago. Suddenly, there's a proliferation of serving frameworks, and there seems like every week Maybe every month there's a new feature that offers performance improvements. So now part of the problem of deploying and maintaining performant models is keeping up. And in some contexts, you don't have to keep up. You deploy something, you have a stable stack, you live with that for a while. But if a new feature comes out and it offers a 10% improvement in performance, it makes your models faster, it makes them cheaper, eventually you're going to want to pick that up. And sometimes that means changing frameworks, and as we'll talk about, that's often more difficult than it should be. So often it feels like this. There's so many options, but it's really hard to take advantage of those options in a frictionless way. So my solution for that is to minimize the cost of experimentation. It's inevitable that you're gonna wanna change frameworks. There will be a feature that you want. There will be a performance improvement that you want. You're gonna have to change frameworks. And so I think, especially if you're starting out to build a stack or you're starting to learn about these things, prioritize modularity and minimize the cost of experimentation. Make sure your stack is at least looking towards a future where it's easier rather than harder for you to change technologies. So how do we make it easier to serve language models? I want to talk a little bit about what makes language models slow. This isn't maybe actionable information, but in the spirit of providing information that I wish I had at the beginning, there's, there's a lot of buzzwords, there's a lot of techniques. Most frameworks use these in various ways. And if you're starting out trying to evaluate frameworks, you need to have some idea of what people are talking about, what's real and what's bullshit. And if you have to learn a bunch of buzzwords and you don't know what they mean, that can be really hard to wade through. And so I think a little bit of low-level information about how inference happens and how we can improve it can go a long way. So what makes language models slow? Turns out it's just two things: memory bandwidth and software overhead. And there's a small set of techniques, methods, strategies that people employ in combination to mitigate these two things. So memory bandwidth. Transformers are fundamentally memory-intensive creatures. Transformers, neural networks, are implemented as a sequence of operations And each of these operations, if you're running on GPU, require transferring data from device memory, the smaller, faster memory caches. That's just what happens. And this is a performance bottleneck. In most cases in production, language models are slow because they have to transfer data from slow memory to fast memory. We fix this by Transferring less data. There are a number of strategies that we can use. Some of you have probably heard of CUDA kernels. They're very fancy. There's another large organization around learning CUDA, and one of the best ways to mitigate the memory bandwidth bottleneck is to just use better CUDA kernels. For example, you can fuse kernels. Kernels. And kernels are just, they're just functions that run on a GPU. That's it. It's just a function that runs on a GPU. You might have a softmax kernel, you might have a layer norm kernel, you might have an attention kernel, etc. Fusion is when you combine these kernels. And so if you combine them in an intelligent way, you can minimize data transfer. That's great, decreases the bottleneck effect. Flash attention is a, it was a very Very impressive. Made a big splash. Still is quite performant. FlashIntention2 is very performant for long sequences. It does kernel fusion, and it focuses on minimizing data transfer. So it's just a better implementation of attention that minimizes data transfer. Paged attention, something similar. More efficient data storage, more efficient data transfer. So these, all these tricks that people implement with kernels, much of the effect of kernel optimization comes down to just doing better data management. And then of course the other thing you can do is make your data smaller, quantize your model. Your weights are smaller, you don't have to transfer as much data, you can transfer more in each shot. We could say that speculative decoding kind of fits in this regime. So there's really There's so many— there's a buffet of optimization methods that you can use, and it can be quite confusing to think about all of them. But fundamentally, much of what they're trying to do is to just minimize data transfer or make it more efficient. The other main bottleneck for transformers is software overhead. So every operation, every kernel has to be launched, and this takes time, has to be scheduled, and this requires communication between CPU and GPU. This is another bottleneck. So the question is, how do we minimize overhead? Turns out that smarter kernels are also a great way to do this. Using kernels means that you have fewer kernels and then fewer kernel launches. So kernel fusion again is really important. Another very common approach is CUDA graphs. So as you conduct a forward pass through your model, you have a sequence of operators or kernels that have to be scheduled, launched, and executed. What a CUDA graph is, is it's a trace across all of those operations, and after you assemble a CUDA graph, you can launch all of your kernels as a single unit. Now there are some caveats. Sometimes you have to reimplement a model so it's compatible with CUDA graphs. But in general, CUDA graphs are a really useful way to make a substantial impact on the software overhead. So these are the two main bottlenecks for Transformers. So this is— Transformers are slow because of memory bandwidth bottlenecks and because of software overhead. And almost all of the Transformer optimization efforts boil down to mitigating these two bottlenecks, often with some combination of the things I just mentioned. What makes language models fast? We talked about some low-level optimizations— kernel fusion, kernel optimization, CUDA graphs. There are things that we can do at runtime also— continuous batching, which I'll talk a little bit about in a second, AV caching. So instead of Having to re-encode, say, your prompt during chat or your chat history during chat exchange, you can retain your KV embeddings and then reuse those for subsequent computations. Hardware upgrades, that's a good one. Just pay more money for your GPU. Your models will get faster. And then there are some tricks. We can call speculative decoding a trick. Make your outputs shorter. That'll decrease response latency. If you can have a successful response that uses fewer tokens, you can do that, and your, your responses will be faster because they produce fewer tokens. To an extent, shorter inputs can also make your models faster. So you have to have a very large, let's say, prompt. If you can decrease that, you might see a small benefit. And this is not so much, not so much a major impact for for small prompts. But if you're doing very, very large context operations, like across many documents, many pages of documents, this can have a big impact. I want to talk a little bit more about continuous batching, though. Continuous batching was introduced, I think it was Orca, close to 2 years, a year and a half ago, ages ago. Before continuous batching, to, to run a batch through a model, including language models, you'd have to assemble each item in the batch. And so people would wait for small periods of time for requests to come in, assuming that you have, um, uh, an inference regime where requests come in stochastically. So you have to pause, wait for some requests, assemble them into a batch, and then run inference over those batches. And that was a terrible way to serve language models. Because one of the weird things with language models is that requests are really non-deterministic. You might have a request that generates 3 tokens and then it's done. You might have another request that generates 2,000 tokens. With the standard micro-batching regime, you have to wait until all of the requests are done before you can redistribute the responses to each, to each each user. Um, continuous batching solved that problem, and it also solved the problem of injecting new requests into a batch during inference. And the way this is done is really quite elegant. Instead of thinking of inference as operating over requests, you think of it as operating over steps. When you're decoding tokens, generating tokens from a language model, really what you're doing is you're running multiple inference steps where each step produces a single token. And if you orchestrate your batching, uh, sort of from that perspective, then you can introduce a new item in your batch and then just decode the next token. And if a particular item in your batch completes, you just pull it out and send it back to the user. And so this fixed two problems. One, you didn't have to wait around for new requests to assemble a batch. You can just continuously batch them. As requests come in, they're injected into the process and then they're, they're executed and when they complete, they're pulled off and sent back to the user. This is great. As you'll see, this adds some complexity for situations where you care a lot about performance too. I'll emphasize here that a consequence of continuous batching is that you can wind up with dynamic batch sizes. So you might, if you have no requests, you might have a model sitting idle. You have no requests, nothing is in your batch. Say one request comes in, so then you have a batch size of 1. Then another request comes in, you have a batch size of 2. Maybe you max out in this particular moment at a batch size of 10, and then one of them completes and you go down to a batch size of 9, and then 4 complete, you go down to a batch size of 10. Then a bunch more requests come in and you go up to a batch size of 25. The point is with continuous batching, you have dynamic batch size. And that has huge implications on the cost of operating your model and the performance that you can provide. So I talked about a bunch of different ways that you can make models fast. Most inference servers these days offer many of these affordances. VLLM, TGI, FastGen, TensorRT LLM, SG Lang, all are really cool, all have different affordances. They all do things like continuous batching. They all have specialized kernels. Many of them use the same kernels. VLLM now uses CUDA graphs. TensorRT LLM compiles to TensorRT, uses CUDA graphs. So while the, the interfaces for these frameworks are a lot different and some of them have support for different features, they all are using many of the same optimization techniques. Where things get really tricky is when you care about performance, like I said before, and there are kind of 3 different ways to think about performance. You can think about total tokens per second, number of tokens produced across all of your requests. Single stream tokens per second is the number of tokens produced per second for a single request. And then requests per second is more of a standard latency measure. How many requests can you complete in a second? This gets really tricky with language models, particularly under the, the regime of continuous batching. Specifically, the relationship between total tokens per second and single stream tokens per second is, is one of tension and I'll show you one graph and just talk through a little bit about what we see here. So on the y-axis, we have single stream tokens per second. On, I think this was for Llama 70B. I don't remember what hardware this was on. This was a while ago that I made this graph. But on the y-axis, we have single stream tokens per second. So this is the tokens per second that a single request will have. So a user sends a request, what's their experienced tokens per second? On the x-axis, we have batch size. Now remember, I said that under the regime of continuous batching, batch size is actually dynamic, which makes it really hard to offer performance SLAs or even think about what kind of performance people get. Sort of perverse consequences that especially, I don't know, like last summer, last fall, still today, there are a lot of frameworks, a lot of platforms that make promises about speeds up to X, and they're talking about single stream tokens per second in most cases, speeds up to 150 tokens per second. And then you make a request against some of these services and you get 40 tokens per second. And that's because they're talking about maximum single stream tokens per second, which also happens to be at a batch size of 1. And nobody's running these models at a batch size of 1, and nobody really has control over what batch size you're going to kind of be operating over with your request. So the performance you get varies substantially because batch size varies substantially. Another consequence is that as you increase batch size, single stream tokens per second goes down, but total tokens per second, which is annotated here as TP, goes up. And so this is why it's really important to think carefully about what your performance SLAs actually are. What are your priorities? Do you need to produce a lot of tokens per second across all requests? If you're using agents, if you are maybe doing something that is not latency sensitive, you have a large batch, it can make a lot of sense to use a really large batch size, prioritize total tokens per second.

I think a good point, Joe. Like, I think a lot of people report benchmarks and they confuse the two. They're like, oh, look at my total tokens per second. And then, you know, they're not telling you what the single stream tokens per second are. And it, they like get really excited. Like, oh, like I have like such great total, you know, total tokens per second, like basically throughput. Um, and so it's really good to pay attention to this. I'm glad you brought it up.

It's, it's yeah, kind of crazy that it's still so chaotic. Um, quite recently we were working with a very sophisticated group that had put together a very sophisticated model, and we were asking them for benchmarks, and they just gave us total tokens per second. And I said, well, what's the single stream? And they had not even thought about measuring that. And then they did, and it was really bad. Um, one of the, the, one of the things you have to think about if you're serving for users So people who are making requests to your model is user experience. And that has to be weighed against the cost of operating your model. As you, as the operator, as the person paying for the deployment, as you increase tokens per second, assuming you have the demand to actually saturate or reach those tokens per second, your cost goes down, right? You pay a fixed cost for your GPU. So if you make total tokens per second go up, your cost per token goes down, which is great. That's what you want to do as the operator. The problem is you make the experience worse for your users. So in the case of this, it was a very sophisticated model developer. It turns out that we really couldn't give a good user experience at these high tokens per second that they were excited about, and we had to cut that down a lot, and that made the model very expensive to run. It's really easy to forget about these details and wind up with something that is a— provides a terrible user experience or is, is prohibitively expensive. And so that's why it's really important to think about these, uh, these criteria during the specification of your deployment. You really don't want to wind up with a model that you can't run because it's too expensive, or that your users won't use because it's not fast enough.

Joe, one question. This is awesome. And actually, if we could stay on that chart, it looks to me like you can pretty tightly approximate single stream tokens per second as total tokens per second divided by batch size. I imagine there can be some variability there, but I've always just reported total tokens per second and batch size together because that seems to approximate that well. Is that the case in your experience?

Yeah, you can definitely approximate it. There's some weirdness. I have some other graphs. I wish I included them. It's very much not a smooth curve. It's like there are a lot of step functions, thanks to GPU architecture. And so it's not perfect. But yeah, I often benchmark like that too.

Yeah, I guess I'm not saying that you can just take your single stream tokens per second, multiply it by your batch size, and scale out to infinity. I'm saying you measure total tokens per second and divide by batch size. To get single streams.

Yep, totally. Yeah. So increasing batch size will asymptotically maximize your total tokens per second. If you are generating synthetic data, if you're not in a latency, latency-sensitive context, there, there are many instances where it really makes sense to just maximize total tokens per second. In that case, you want to use a really large batch size. I say asymptotically because at some point you will reach the memory bandwidth threshold and you're just not going to get a return. Adding a new unit to your batch will just sort of linearly increase the time that it takes to complete. So there's a limit to the gain you get by increasing your batch size. Decreasing batch size increases single-stream tokens per second, but it penalizes total tokens per second. So if you care about latency and you want the fastest possible tokens per second at the request level, you should run a batch size of 1. Nothing will be faster than that, all things held constant. That's also the most expensive way to run your model because you have to scale horizontally as you get concurrent requests. The right balance depends on your use case. Thinking about your target batch size is really important. And it has to be tuned to the particular context that you're playing your model in. Okay, so we talked about making models faster. I wanna talk a little bit now about simplifying language model deployment, specifically thinking about making sure your stack is modular so that you can do experiments across different frameworks. And then we're going to do a quick demo on Replicate. So I think it's really important to prioritize modularity. Different frameworks have different affordances. I haven't told many of our stories, and I think so many of them fall into this category. Just recently, I was working on speculative decoding with trt-llm, which has been released since January. That's several months. TRTLM is in development. It's pretty well documented. Sometimes there are bugs, but in general, I really like the framework. I think they're putting a lot of great effort into developing it. I thought that speculative decoding would be an easy win, except it didn't work with streaming, and there were a few other things it didn't work with. So it, it, it worked sort of, but not for my use case. And that was really frustrating. Time and time again, I found a feature that I needed wasn't implemented in the framework that I wanted to use, or it wasn't compatible with other features I wanted. For instance, TRTL now supports sLoRA, but not with weight-only quantization. So you can't use qLoRA. And we were serving many of our models in 8-bit, and we couldn't use sLoRA with that. And that's, that's kind of, that's just an implementation detail. There's no reason for that to be the case. That's just where trt-lm is in its development phase right now. So I think it's really valuable to be able to change frameworks as you need to. To do that efficiently, you need to be able to experiment with different frameworks. As often as I've found that a particular framework was lacking, I've also tried to switch to a new framework and then found that a particular composition of features weren't compatible. So maybe I'm using features X and Y in my current framework. My new framework has feature Z as well as X and Y, but it doesn't support the union of X, Y, and Z. It's better if you can figure that out really quickly rather than having to sink a bunch of engineering effort into it. In a perfect world, these things would all be documented, but this goes back to the, the problem of a proliferation of new technology. People are constantly introducing awesome features that we should all take advantage of. But that means that there are bugs and they're not well documented and they don't always play well with other features. And so figuring out how all these pieces can fit together in your stack is honestly one of the biggest challenges to deploy language models on the bleeding edge. And that's one of the problems we're trying to simplify on Replicate. So Replicate is a serverless infrastructure. We offer ready-made APIs for models. You can run any model on Replicate. But we also have a piece of software Hamel mentioned called Cog. And you can wrap your model, your model serving code with Cog, and push it to Replicate. And one of the implications of this is that it gives you complete control over your serving framework. We're serving our kind of official language models, uh, with TRT-LLM right now, and specifically a cog implementation of, of TRT-LLM that we're gonna open source. We're also serving some with VLLM. And so the direction we're going with Replicate is to make it a place where not only do you have all these models to experiment with, also have a bunch of serving frameworks to experiment with. And you're still gonna have to get into the details if you care a lot about performance tuning features that aren't supported out of the box or might have to be activated. But, but my goal, my kind of personal goal, is after all the blood and sweat and tears, is I think it should be a lot easier to experiment with these frameworks, and it should be a lot easier to figure out what's broken and what isn't. So right now we're close to open sourcing CogTRT LLM, and we did an enormous amount of work last week to be able to open source CogVLLM, which which is what I'm going to show you. And the idea is that you can use these servers as a drop-in solution, and we want it to be really easy. Sure, you want to run a model with the LLM on Replicate, very easy. But perhaps even more exciting is that you can just pull a repo and change whatever you want to change. You can change the inputs and get a different API signature. You can add support for a new feature that perhaps we haven't implemented yet in Coq-LLM. And the idea is to actually take advantage of the fact that Replicate is completely open source. The way we serve models is open source. You can look at how we do it, learn from it, and if you're really nice, open a PR and make it better. So I'm going to show you our VLLM walkthrough and walkthrough workflow workflow and talk through some of the Replicate details. If anything isn't clear, please stop me. And yeah, now switch to browser and I'll show you. Okay, let's start with So when I go to Replicate, I land on my profile. There's a dashboard, shows you recent predictions, recent trainings, recently pushed models. You can look at your models here. You can go to an explore page and see all of the models that are available. You can go to these models, you can see a UI that allows you to run things against these models. But Replicate also provides APIs, clients. So there's a Node client, there's a Python client. Replicate makes it really easy to kind of drop in support for any particular model that's running on the platform. What I wanna show you is something we set up that makes it really easy to, to run a model with VLLM. This is currently implemented as a, a training on Replicate. And we're working on introducing some new abstractions that'll be a little bit more coherent and elegant. But on Replicator, training is really just something that runs in a container that produces some artifacts, some weights, and those weights are then associated with a downstream model. So we set this up as a training so that you can pull weights from Hugging Face, and then those weights get associated with our VLLM server. And what you get in the end is a replicate model running a particular set of weights, particular architecture with VLLM. And I already did these steps. It takes time to download weights and transfer them to, to replicate. I'll just show you kind of what you would do if you're doing this yourself. We have also this documented. So we put in a name. We would add the Hugging Face ID. I'm doing Gemma 2B Instruct right now. You can add the model SHA if you want, which allows you to specify a revision. And I'm going to drop in a Hugging Face token. Here, which is registered as a secret. And this is really right now just a, a wrapper around the Hugging Face hub. So it has all the affordances that you would expect. If you want to disallow files associated with the, the model Hugging Face, you can have an ignore pattern, allow pattern, et cetera. You can also specify a prompt template here. Otherwise, we'll use the default chat template associated with the model's tokenizer. So you click create training, and that will take you to a page that looks like this. And what happens is we pull the weights from Hugging Face, we put them in a tarball, and then we push them to GCP. This is important, and we're encouraging people to shift towards workflows like this, because if you put things in our buckets, we're able to cache them and offer much better downloading performance. And we have a several other optimizations that we're introducing soon, um, that will help a lot with cold boots. We regularly see people put models on Replicate that, for example, just pull in a naive way from Hugging Face. That's a really slow way to download your weights. That's a good way to make your cold boots even worse. And so we're working on affordances to make it really simple, or rather really hard to do some things suboptimal. You shouldn't even have to think about these problems. That's, that's kind of where we're trying to go. And so right now, to get something running with VLLM, easiest thing to do is to just go to this form, and this form will allow you to transfer weights from Hugging Face to Replicate infrastructure. And so it downloads, pushes, and then when the training is completed, it instantiates a new model. And so we could click run the trained model, and that would bring us here, and this is our model, and we can make requests against it. And as I showed you before, um, there's support for different clients. So if you have a Python client and you want to use this model, you have an example here how to do that. So what we did was we took just Gemma 2B instruct from Hugging Face, and we're now running it with VLLM on Replicate. Where we're going is you can do this with SGLang or TrtLLM or any of the other frameworks out there and get the affordances from those frameworks. The other part I wanna show you is cog_vllm. And so this is the VLLM code that actually makes this all possible. And the process that I just walked you through is all documented here. So you can do this yourself. But if you want to do some local development, you can follow these instructions. So you'd install cog, Hamel mentioned before. You'd clone this repo. And then you would run something like this. So you would set cog_weights variable. And you would set this to the URL that we give you when you complete your training. And I'm going to go ahead and run this in VS Code. This. So I've set my cog weights to this URL. So this is an object created by the VLLM creator that I showed you. So that creator takes weights and other artifacts, including the tokenizer from Hugging Face, puts it in a tarball, puts it in one of our buckets. You get a path back. Now you can use this path for really fast downloads, replicate. So I set this environment variable and then I just call predict and it's going to be really fast because I've already downloaded the weights. Um, so if I hadn't downloaded the weights, they would have to be downloaded locally and they'll be cached. We see the VLLM server starting up and soon we're going to see some tokens emitted. Um, so let's see, we'll see something about CUDA graphs in a second. Is it's model agnostic. That's something we alluded to. So one thing you have to think about with model serving is support for models. That's new for— so you might pick a particular framework, and then a new model comes out, and you want to use it, except it's not supported. And that's really annoying. VLLM is great because it pretty much has out-of-the-box support for Transformers. TRT-LLM has focused a lot on adding support as new models come out, and they often collaborate with major model providers. Like when Gemma dropped, there was support immediately. But it's not always the case, and being able to exchange serving frameworks can kind of, can really save you if you need support for a new model that's not supported in your framework of choice. Here are tokens. The model responded. This is a list of strings because we're streaming. And this is a model run locally on a VM. You don't need a VM to push, so we could go ahead and change some of our code. We could modify how inference is done. We could push that to Replicate and run there if you don't have a GPU available. And then very quickly, I'll just talk through some of the components that Hamel Husain already described. So we have a predict.py that's basically doing most of the work for us, and we have a config YAML or a cog.yaml. The thing I want to draw your attention to here is this concurrency argument, and that's what allows the model to do continuous batching on Replicate. And so you can specify the maximum concurrency that Replicate will, will send to your model. So here we've set it to 32. That means Replicate will never schedule more than 32 requests against this model. And it'll start to scale out as you reach that max concurrency. And internally, what we do right now is we target about half of the max concurrency as the target batch size. But as you're thinking about performance SLAs, that's something to keep in mind when you're thinking about how this will kind of scale out and operate under production loads. So if you're targeting 16 requests, you can think about if you have that at equilibrium, what will your performance be? How bad will your performance be if you reach the max batch size? One of the benefits of continuous batching is that you have a lot of flex, so you can support dynamic demand without having to scale out. But the cost you pay is that you also have dynamic performance. So if you get a big spike, bunch of requests come in, everybody gets worse performance. Um, I'm gonna stop here and see if there are any questions.

Probably have to move, move on, uh, from questions just from time management perspective.

Good.

I'm always, uh, carry this into office hours or something like that.

Okay. I, I just got a message from Travis. I think he's got a, gonna have a hard stop at some point. So I think we probably need Travis to go next.

Awesome.

Yeah, thank you, Dan.

Thanks, Joe, that was really good.

Thanks.

All right, should, should I go ahead and share my screen, or—

Yes, please.

Yeah.

All right, okay, thanks everyone. Um, yeah, so really, uh, enjoyed the last two talks. Here, um, think that set the stage really well for what I wanted to talk about, which was some very specific nuances of deploying fine-tuned LLMs in particular. I think, uh, one thing that, uh, I've focused on a lot in the last year or so has been this intersection of fine-tuning and serving and what unique challenges or opportunities even can be presented when you're trying to serve fine-tuned models in particular. As opposed to base models or, you know, merged fine-tunes into base models. So the main theme of this talk will be lessons learned building our platform for training and serving fine-tuned LLMs. Just as a little background about me, so I'm the co-founder and CTO of Predabase. We've been around for about 3 years, 3 to 4 years now. We were originally founded out of Uber's Michelangelo ML platform team. I worked there as a lead on the deep learning infrastructure team. I was there from 2016 to 2021, and while I was there, I was the lead maintainer on a project called Horovod. It's a distributed deep learning framework for PyTorch and TensorFlow, if folks remember TensorFlow, and Ludwig, which is a declarative deep learning framework that actually has a lot of similarities to Axolotl, but we built it back before LLMs, so it was much more targeted towards image classification, text classification, use cases like that early on. And then most recently, Lorax, which is our framework for doing fine-tuned LLM inference, which I'll speak a little bit about today. This is the most salesy slide that I have, which is talking a little bit about our platform itself. So Predabase is essentially a managed platform for fine-tuning and serving fine-tuned LLMs. We try to be very integrated end-to-end in terms of providing the ability to prompt models out of the box serverlessly, fine-tune them, and then deploy them either serverlessly or dedicated in VPC or SaaS, etc. And so the rest of the next few slides will be talking a little bit about some of the specific challenges that we had to overcome when building this platform and how these might relate to challenges that you're likely to encounter when serving fine-tuned LLMs as well. So let's talk a little bit about our story then. So our key kind of thesis on the market is that these very general-purpose LLMs like ChatGPT, etc., are great. But, you know, when you're trying to put something in production, you want to put something specific into production to solve a specific problem. And so the joke that we like to say is that, you know, general intelligence is great. I don't need my point of sale system to recite French poetry, which is basically saying you're paying for that extra capacity one way or another, whether that's in dollars or latency, etc. And so ultimately getting fine-tuned models in production is all about getting something more specific to your task that is better suited to it and better optimized for that. So the common pattern that we believe that organizations will go through is starting with something like GPT-4 as a starting point for experimentation, and then over time migrating to fine-tuned task-specific models where you might have, you know, a model for your customer support tickets, one for sentiment analysis, one for chat, et cetera. And so the future that we envision is you have lots and lots of fine-tuned models for all your different business tasks and use cases. But a challenge quickly comes up, which is, okay, if I'm going to fine-tune a new model for every task, how much is that actually going to cost, uh, to serve all those models? And, you know, if you just kind of look at like a standard entry-level GPU in AWS, like an A10g, running about $1.21 on demand per hour. It has just enough VRAM to serve some of the 7 billion parameter models nicely. Quickly, you see that this starts to accumulate, and before too long, you know, you have 16 different use cases, you're paying, you know, $14K a month in cloud bills to serve those, and it only gets worse from there, right? And so this was the same dilemma that we faced when we were thinking about building our platform because, you know, we wanted to build this serverless platform for people to fine-tune and serve. But if every single user that came in needed a new dedicated deployment, uh, even when they're just getting started, this was going to be very expensive for us. And so taking a step back and thinking about how you deploy fine-tuned LLMs the old-fashioned way, you know, you normally have something like a Kubernetes pod. You have a request queue coming in that, uh, is, you know, accepting requests and queuing them up for completion. And then you have your fine-tuned model weights. And if you're doing, you know, parameter-efficient fine-tuning like Dan mentioned at the beginning, these weights only account for something like 1 to 5, maybe 10% at worst, of the parameters of the model. But when you're deploying all these things, you're deploying them over and over again with those same base model parameters replicated each time, right? So the observation you might have is, well, what if we just took these base model parameters and then tried to serve these fine-tuned model parameters together on top of the same model deployment? And that was exactly what led us to want to build this framework called Lorax, which is a fine-tuned inference system. It's built on top of Hugging Face's TGI, and since then we forked it, expanded a bit, and now use it to support our serverless inference on Predabase. Um, and essentially it's that idea, like, let's take all these different requests coming in, uh, for all these different fine-tuned models and try to serve them concurrently on a single deployment, provided that they all share a common base model. Uh, now you might imagine that naively you could do this by just swapping these adapters, but you might realize that this would be quite slow. So what we do instead is we batch together different requests for different adapters at once at a single time. So you might imagine having requests for an adapter in blue here represented, adapter in red, and then maybe gray is, you know, another adapter. And then instead of trying to do them one at a time, what you can do is kind of logically think of them as being fuse together into a single tensor for the A's and a single tensor for the B's, and then just do one straight shot, you know, multiplication against the base model parameters and then multiplications against the A and B matrices. And this actually is, you know, much better than doing it sequentially, but you can do it even better by lowering this down to the CUDA level so that you're not having to actually physically join, or rather gather all these LoRAs together into these giant A and B matrices. Instead, you can do this down at the CUDA level using indirection through pointers and, you know, taking advantage of different tiling strategies and do it much more efficiently, which was some work that was done by folks at University of Washington on a paper called Punica back in September of last year, which was in the basis for like SLoRA and various things like that. And what you find when you do this type of approach, which we've added in Lorax, is that if you compare against the baseline of trying to do every adapter individually, you see that as your adapters increases, the throughput starts to drop pretty precipitously. You know, by the time you've got 125 adapters, then your total throughput, um, so 128 adapters, your total throughput has dropped, you know, by about 90% or so. But by doing this, intelligent, you know, fusion heterogeneous batching of different adapters together, we can maintain that peak throughput with only about a sub-10% degradation even at very extreme ends of the number of adapters. And so that's, you know, really the key, you know, initial hook about Lorax that's, you know, made us want to put this out there is having a production way to be able to serve all these things concurrently. At once without degrading the performance for, for our end users. And ultimately, we believe that this translates into, you know, the bottom line being cost savings, right? So if you compare against fine-tuned GPT-3.5 as a baseline, um, you know, I don't know if these numbers are the latest up to date, but last time I checked, about $6 per million tokens for the fine-tuned models. Um, they have that as a constant because it's serverless. But if you're comparing that against dedicated, dedicated, of course, is going to scale up as you add more and more replicas for different use cases. But the idea with Lorax is that you get the scaling of something like serverless, but with the baseline cost of dedicated at very few replicas. And so overall, the argument is that you should be able to see pretty dramatic cost savings by doing an approach like this. And so that was ultimately what we ended up doing and served as kind of the foundation for how we are able to make our platform work the way it does. So now I wanted to kind of move off of a little bit, you know, talking just about us and how we got here and more about, you know, some practical advice for you as well. And so one of the things that, you know, users come in, they're interested in this kind of capability. One of the first things that happens is they say, okay, I trained an adapter. For my particular dataset, now what? What happens next? I want to revisit this concept of merging that Dan and Hamel spoke about earlier because I think it's actually quite important as the first step is like, okay, what am I actually going to— what's my serving strategy going to be for my fine-tuned model? The case for merging is pretty simple is that you get good baseline performance when you merge the adapter because you don't have to pay for any of the overhead from processing subsequent layers at runtime, right? You just merge those LoRAs back in, and so that effectively eliminates all that computation, which is obviously a win. But that in and of itself is not the only consideration. Other considerations might be, do I need to serve other models as well, right? So if you want to serve multiple fine-tunes, or even if you want to serve the fine-tuned model plus the base model, that's a very good reason why you might want to consider not merging, uh, since then you can pack all those together in a single deployment, right? Additionally, I think one thing that came up previously, as well as in the comments, was how does this interact with quantization? And it turns out that the idea of merging the adapter weights back into the base model, uh, gets very tricky in a world where you fine-tune the model using QLoRA. Uh, the reason, because you're no longer serving something that is FP16, those weights are intrinsically tied to the quantized model, whether implicitly or explicitly. Additionally, we need to think about our iteration velocity. I think someone else mentioned previously that things change very quickly. New models come out, new datasets come out. You want to be able to very rapidly A/B test or verify experiments. Being able to kind of hot load adapters in quickly is a very good way to, say, run an A/B test or do some shadow traffic before promoting your model to production to say, does it actually perform better on live data without having to, you know, double your serving costs just to be able to run that test. And a couple final notes is it doesn't work on all adapters. So LoRAs, it certainly does. Others like DORA would be more complicated. Other adapters that I'll talk about, like speculative decoding adapters, you can't merge them. And finally, it adds to additional disk space. So all in, it is a consideration, but, uh, it's one that I think needs to be done on a case-by-case basis, is whether you keep the adapter in its unmerged form or merge it back in the base model. Um, and I want to talk about this quantization one a little bit more depth because I think we see this come up all the time with our users, and I imagine it's something that you're likely to encounter as well. So let's say you've fine-tuned a model using QLoRA. How are you going to serve that model? I think one of the first things that people think of is, well, I fine-tuned it in quantization because fine-tuning requires more memory because of the gradients and everything else. But I'm going to serve it using full precision or FP16 half precision because I want to not have to pay the cost of dequantizing. And the memory overhead is less, so, at serving time, so I can generally get away with it. However, one thing that you're likely to discover is that the weights, the activations that come out of that base model portion after dequantization from QLoRA and the activations that come out of the base model in FP16 are not actually the same. And that's because there's some amount of error that exists when you quantize and dequantize. And so as a result, that adapter was actually fine-tuned in such a way that it's connected, right, through noise, really. But like, it is connected to the fact that it was trained on that quantized adapter. And so therefore, just swapping them out is going to produce different behavior and may actually degrade performance substantially. So the alternative might be, well, what if I just served it in the QLoRA quantized form since that was what I trained it on? And you can do that, and that would work, and Lorax supports it. You know, you can have the adapter be used with the quantized base model at runtime, but the trade-off there is that performance is significantly worse in terms of latency and throughput when you're serving the quantized model. So here you can see time to first token. This is just running on an A100 with like a Mistral 7 billion. You can see that Time to first token with FP16, you know, significantly lower, less than half of the bits and bytes. And then even on throughput, so tokens per second, this is single request tokens per second, you can see a pretty significant drop there as well. So if you can get away with it, you definitely would rather not serve with QLoRA. So that presents the dilemma is that serving on FP16 produces worse results. Serving with bits and bytes quantization is slow. Is there a way we can get the best of both worlds, quality and speed, for serving these models? And so that's where a little trick comes in, which is what if you dequantize the QLoRA weights? And so this is something we actually do quite a lot with our customers is you take the original weights of the model in FP16, quantize them using bits and bytes as you would for QLoRA. Now you have the weights in NF4. And then you just reverse the quantization because you need to do that at a per-layer level anyway for QLoRA. So you have the functions there to do it. And the result is a set of FP16 weights that is now numerically identical to the quantized weights. And then now you can go ahead and serve that in FP16, get all the performance benefits of doing so, but with none of the performance degradation that you had before. And so in practice, this is what we end up doing the most. For users who fine-tune with QLoRA. A couple notes on performance. So I think this one has been a lot of good discussion previously about performance optimization for base models, so I don't want to spend too much time on that particular note, but definitely one thing that I always mention to customers when we're getting started with a POC is, you know, what is your data and load requirements in terms of queries per second or requests per second? What does it look like in terms of the distribution of input and output tokens and the number of adapters that you intend to serve in production? Certainly, when we talk about ideal batch size, target throughput numbers, things like that, they vary quite dramatically depending on the combination of these variables. So for example, high requests per second with a high number of input tokens, you know, you're ultimately bound in terms of how many how much throughput you can get by the prefill step in a large way, because that is the bottleneck of how many requests you can pack into a single batch to then serve to the model. So if all your requests are very small in terms of input tokens, you can increase the parallelism quite significantly and therefore increase your total throughput quite substantially. Whereas if all your requests are very long context and you're generating a few number of tokens, your number of output tokens generated at peak is going to be much lower than it would otherwise. So those things all have a very big effect on what numbers you're likely to see or what you should set as your expectations. And then apart from that, you know, that then of course leads into what your service level objectives, your SLOs, are. So some users are very sensitive to peak throughput, total tokens per second, usually for batch use cases in particular. Where you want to scale up, process everything, and scale it down. Maximum end-to-end latency can be very important for real-time use cases, of course. And then cost per day, per month, per year. I think when you're using open source, one of the main reasons to do so is to reduce costs relative to commercial APIs. And so I think that one comes up by far the most and is intrinsically very tightly connected to peak throughput as well, because if you get higher throughput, you require fewer machines, or you can do it faster, so they have to be up for less time, and therefore you can reduce costs. One other thing I think is very important for deployments is thinking about, um, what hardware you need to run your deployment in the first place. Uh, so as a typical recommendation, I would say back of the envelope, at least 1.5x, uh, the model weights are needed for serving, um, the model. And that ultimately comes down to the fact that it's not just the model weights that are required at runtime to— that occupy memory on the GPU. You also have the activations, which is intense, uh, intrinsically connected to, uh, things like how many tokens you're allowed to fit in the batch at a single time, how much— how long of an input length you're going to tolerate per request, um, the adapters as well, how much memory you want to allocate for serving different LoRA adapters if you're using Lorax to be able to support multiple at a single time. And then of course the KV cache itself, uh, which is going to need to be bigger if you're generating more tokens and supporting larger batch sizes, since all that needs to be kept around for the duration of the entire request in order to avoid having to recompute, um, the attention between different tokens. So all these things add up collectively and need to be considered when choosing hardware. But to me, the key questions that I think are worth asking are always, you know, okay, given this, how much VRAM do I need? How many requests per second am I expecting at peak? What are— how are the requests distributed throughout the day? Like, are they all concentrated at once, in which case, you know, I can scale up, handle it all, scale it down, or are they distributed pretty evenly? What's my maximum acceptable latency per request? Is it something where I can let it complete in minutes or hours, or does it need to be milliseconds? Am I willing to sacrifice quality or latency to reduce cost, so willing to use quantization? And how many different tasks do I need to support in production, so things like whether I need to use multi-LoRA inference. And then last note here, I think, in particular for us since we support both serverless and dedicated deployments on our platform, one question that gets asked a lot is, should I use serverless or should I use dedicated? And to me, it comes down to the same set of things that were discussed below, but then how that translates into the requirements for your use case. So when request volume is low to medium but distributed fairly uniformly, so you're getting like some set of requests coming throughout the day, you know, you're okay with latency on the order of seconds, like some amount of variance in that, that to me is like the perfect serverless use case. Because it's not overly sensitive to a specific latency requirement, but it does need constant uptime. And so that's one of the cases where Dedicated tends not to be the best option because with Dedicated Always On, you're paying for it whether it's utilized or not. And that's definitely a bummer if you're not getting a lot of utilization. But Dedicated does become very attractive even though it seems like serverless costs are so low when you have one of two scenarios where either the request volume is very high, concentrated, and spiky, so, and not real time. So something like batch where latency is, you know, minutes to hours, but request volume is very high within those time windows. Or on the opposite end of the spectrum, when request volume is high and consistent latency and throughput SLOs are critical, so you need millisecond to second at most, latency on these requests, and you have lots of them coming in at once, and therefore the noisy neighbor problem of serverless could result in situations that are not acceptable. And then last thing to comment on before kind of closing out is just a little bit of thoughts on where things are headed, in my opinion, with fine-tuned— with fine-tuning and serving of fine-tuned models in particular, and this notion of fine-tuning for throughput. So there was a little bit of discussion before about speculative decoding, and I think one thing that we think a lot about with fine-tuning today is quality, like I want to fine-tune to make my model better, but we don't spend a lot of time thinking about fine-tuning for speed. But I think that actually there's a huge opportunity there to think about fine-tuning as a way to actually improve model performance from a throughput and latency perspective as well. And to motivate this, let's look at a couple quick numbers of today, like the baseline model throughput, single request tokens per second, compared to running a single LoRA adapter. And so it's pretty well established that there's a little bit of a performance hit you take when you serve a LoRA model. Here comparing just VLM and Lorax for the purposes of kind of, you know, saying it's not just one framework that has this, this issue. So about a parity on base model performance, and then you'll notice that there's a bit of a drop when we introduced the LoRA, right? But there are ways that we could potentially change, flip this around a bit, and actually think about the LoRAs being better. And one way is through thinking about fine-tuning for speculative decoding. So Medusa, for example, is a famous solution that came out last year for fine-tuning additional projections that allow you to predict future tokens and not just the current token. and additionally introduce a verification step that ensures that you're not accepting tokens that are categorically incorrect, that you're always only accepting tokens that are correct. In the interest of time, I won't go into too much details on how this works, but essentially just making sure that the tokens that the model would have originally predicted are in fact the ones that you accept versus the ones that the model would not have predicted are rejected. And as long as you are guessing correctly more than you're not, you'll speed things up. But crucially, there's an opportunity here to be able to combine these different strategies, quality and performance, by fine-tuning adapters that do both at the same time, which is exactly what we've been investigating. Broadly, we call this strategy a Lookahead LoRA, where you want to fine-tune a task-specific model that predicts not just the next token but maybe the next n tokens. And interestingly, we found that this not only degrades performance— it doesn't have any degradation performance. In some cases, you can get lucky, and it might actually do better without tweaking the hyperparameters too much. But the performance in terms of throughput is dramatically better in this case, as much as 2 to 3x what it is of the base model as well as the adapter itself. And this is comparing performance on different use cases here, MagicCoder, Here, conlpp, conlpp being pretty easy 'cause it's a JSON extraction use case as we framed it versus Magic Coder being code generation. But as you can see, there's quite a big opportunity to be able to combine these two techniques in a way that not only helps with quality but helps with throughput as well. And so this is exactly what we've been doing at Predabase. And just as a very quick demonstration of this, I hope this won't take more than a minute, But let me just quickly show a demo of how this works in practice. So here I have my OpenAI client connected to my Lorax instance, and I'm just going to run this demo real quick, see how quickly I can generate some text for this magic coder use case. You see I get 98 tokens per second because in this case I'm using a generic Medusa as the base model. Actually, And now let me go ahead and try a Medusa that was fine-tuned on this data but not with LoRA. And you can see that now performance has gone up to 113 tokens per second in terms of throughput. And then finally, let's try using one of these new fancy lookahead LoRAs that I was talking about where we fine-tuned both for the task as well as for the throughput as well. And you can see that suddenly our throughput has shot up to 147 tokens per second, which was what I showed you in the original graph. So this is, I think, where things are going, in my opinion, is with the intersection of fine-tuning and serving is, you know, let's not just think about fine-tuning as only a quality proposition, but also being an opportunity to tailor models specifically to specific tasks for performance, you know, quality, throughput, and latency all jointly at the same time. And that's it. I'll go ahead and hand it over to Dan and Hamel Husain and Charles for the next bit.

Yeah, I'm going to hand it quickly over to Charles. I know a little bit about what Charles is going to talk about, so you guys should really stick around. This is gonna be awesome. Um, we are planning to run over by quite a bit, and, um, hopefully people don't have a hard stop, but, uh, you really don't want to miss what Charles is going to say. Yeah, we love Charles and we're gonna let him go.

He's gonna say— he's gonna—

it's gonna be epic right now. Um, wow.

Uh, so absolutely zero pressure.

On this, given how epic those other talks were. All right, so yeah, you've already heard a lot of stuff about deploying LLMs today. I wanted to talk a little bit about deploying LLM services on Modal. I was also told to cover batch versus streaming, but it seems like we've pretty much nailed that one. And so I'm just going to give a little bit of what I was going to talk about and focus just on a very high-level takeaway that I want to make sure is very clear for folks about not just like batch versus streaming for language models, but the fundamental tension of throughput and latency. So, um, like when you talk about whether some— like people will sometimes say a software system is slow with no further, um, like explanation. And sometimes that means that it took a long time to do a big job, and sometimes that means it took a long time to do something that seemed very small. And in the former case, it's that the system does not have sufficient throughput. It does not complete sufficient requests in a given unit time. In the latter case, it's that the system has insufficient latency. It does not complete a single request in a certain unit time. And with both of these, when you are optimizing these two things, as we heard already from Joe and Travis, you can deploy resources to achieve different service levels on throughput and latency. And cost becomes the hidden third feature of systems that determines their throughput and latency. So we've already heard all this. So with throughput, we're thinking about batch-oriented LLM inference. So you're refreshing a recommendation system every night or every week, like Spotify Weekly. You're doing your evals and your CI/CD. With real-time, that's things like chatbots, Copilots, audio or video chat, guardrails where an LLM is checking another LLM in production. And so those are places where you're going to have tight constraints on throughput in the first case or on latency in the real-time cases. And where these will become most challenging for cost, where it will be most difficult to exchange cost for throughput or latency are cases where you have consumer-facing applications where standards are very high and like customers are, are very fickle and ready to change and at large scale where some of the things you can do as tricks at small scale fail to continue to develop. So the— what are the constraints on these features of a system? With throughput, you're generally dependent on the up or downstream system, how much throughput can they handle, how much throughput do they want. So if you're producing tens of thousands of tokens per second total, and you're putting that through a logging service, you might find that some of like software 1.0 logging services expect, do not expect tens of thousands of tokens per second, and that logging software will start to fail. If you have CI/CD and you have a bunch of other parallel jobs running, you may not need to have tens of thousands of tokens per second to finish in the, like, 5 minutes that it takes for some other component, like a Docker image push, to complete in your CI/CD. So that's the primary constraint determinant on throughput. It's the systems to which this LLM and its inference is connected. For latency, almost all the time, when we're talking about low-latency systems with LLMs, It's human perception. Human perception of latency is what matters. So first, that means you can use tricks to get around latency constraints, like showing intermediate results or producing a draft that you then replace with a higher quality result. The second takeaway is that the constraint here on latency is at a— on the order of a couple hundred milliseconds for the, for the entire system, not the LLM, but the entire system. So any network calls, any, well, like file I/O, all of that, as long as it fits, this is backed by sort of like psychophysical and user research, as long as it fits in the order of a couple hundred milliseconds, the human actually becomes the latency bottleneck. They're like our ability to react to information presented by the software system. And then finally, on how much cost, how much money do you have to deploy to solve these two problems, it's going to be dependent on how much value you can deliver. So the more value you can deliver, the more, like, the more money you have to deliver that value. So the reason why this, like, the very high level outside of just LLMs, but in general why this becomes really challenging is latency-sensitive applications are some of the most exciting applications of LLMs. It was ChatGPT that got people extremely excited about this technology, and that was one of these chatbot latency-sensitive situations. And it is a fundamental basic fact of engineering that latency lags throughput. This is a famous paper that— written back in maybe the mid-2000s about how much easier it is to improve bandwidth than it is to improve latency. So on the right here, I have a figure of the relative improvements on a bunch of different components of the stack, network, disk, memory, the actual CPU. Over time, the bandwidth improvements have been like super Moore's law, whereas the latency improvements have been much slower. So that like kind of mustard brown line in the bottom right, that's if latency and bandwidth improved at the same rate. All of these lines would be on that. And this is log scale, by the way. So we're seeing that microprocessors got— if you use their full bandwidth, they got over 1,000 times faster during this time period. But their latency, if you're just looking at how quickly can one microprocessor process, say, a single instruction, that only got on the order of 10 times faster. And in general, you end up running against literal physical limits. The speed of light in networking or the, like, amount of heat that gets stuck in silicon when we run electrons through it when it comes to the speed of microprocessors. And you can't bribe physics, but you can spend more money on things to create greater bandwidth. Maybe the most important example of this, the most relevant one for people working on the things in this class, is actually GPUs are a throughput-oriented design for a processor, where CPUs are a latency-oriented design. Substantial amounts of silicon area are devoted to caching and control flow in red and orange on the left-hand side of this figure, and much less to operational throughput, which are the ALUs in green in the top right.

This—

that's the— the CPU architecture is oriented to retrieving small bits of information really fast so that they can be operated on quickly. A GPU, on the other hand, is a throughput-oriented design. Where there's a tremendous amount of silicon area given over to processing and much less to control, but still quite a lot over to caching and in particular to very high bandwidth caches. And so even though, as Joe correctly pointed out, memory bandwidth is the— which is memory throughput is one of the limits on GPUs, they have substantially higher memory throughput than CPUs. And so this, this, like, distinction that has created, like, a new multi-trillion-dollar company in NVIDIA is another example of the, of throughput being able to, like, move ahead of latency over time. This figure comes, by the way, from the, the CUDA book, Programming Massively Parallel Processors, which I can highly recommend to folks. So It's a very— it's a general phenomenon. It's particularly relevant with GPUs. So it's very punishing in LLM inference in particular. We've already seen this. If you want more throughput, you can just increase the batch size. There are penalties to latency, and as Joe pointed out, some penalties to throughput for an individual user. But you see this basically linear scaling in your throughput up until you reach compute boundedness, at which point you can get more GPUs. and continue to basically linearly increase throughput up to the point where you're spending literally billions of dollars on inference. You can just keep on, um, increasing throughput until you start to run out of power plants. Um, but if you want shorter latency, you can basically just go die. Um, so we heard about some of the strategies for this for, for Joe. Joe decided not to just go die and to try and solve this problem. Uh, so quantization, we also heard about this, from Joe that can like, like reduce your latency if done correctly, but it also improves your throughput. You can distill models down to a smaller size. You can truncate them and fine-tune them. That also improves throughput, turns out. So it doesn't just improve your latency. It maybe mostly improves your latency by boosting throughput. You can buy more expensive hardware. If you're running on A10Gs, you can upgrade to L40s. You can upgrade to A100s. Or H100s, they do have faster latencies, so you might get some additional latency that's not throughput related, but you will also improve your throughput at the same time. Or you can do some of the things that Joe was talking about, which is write some really, really hard to write software. So hand-composed CUDA kernels can like reduce latency and by like avoiding round trips to memory and things like that. This turns out will also improve throughput. So if you actually want to improve latency, like you want the shortest latency and you, and you don't care about throughput, there's basically one solution that people have found, which is to run the system entirely on cache memory or what's, or SRAM, static RAM. So that's like the, going back to our little chip architecture picture here, it's like take this, this cache memory here, which is not the RAM that you're used to thinking about. This is the L3, the L2, or the L1 cache in your CPU. And just build 80 gigabytes of that and run the language model off of that memory directly, rather than running it off of the more typical dynamic RAM architectures that, that you're used to thinking of as the RAM in which the memory of the system in which your model weights are stored. So if you want to do that, you should be Grok, who built the LPU, which basically runs off of that, that core principle. There's more brilliant stuff in there, but the core, like, thing that enables them to operate at extremely short latency is to just have 80GB of SRAM to run models off of. This does actually have a penalty to throughput, and in particular, it's such a big change in the system, you actually have to talk about throughput per dollar, you have to connect it back to that third constraint on our LLM inference, which is the amount of money and resources we're willing to expend on these other two features. So if you check out, there's some great Semi analysis post estimating the throughput per dollar of something like Groq versus a more traditional NVIDIA many H100s approach to serving. LMs at scale. So the, this third constraint of cost is one of the, it's like kind of lying there secretly whenever we see a throughput or a latency constraint. It's like, well, we add more money, we could do this. So that's bad news, except for people who can raise $6 billion on a Series A. But for everybody else, the good news is that costs are in fact falling. Over time, we've seen like rapid decreases in price. So this chart, there's a lot going on here. I'm going to try and walk you through it in the time that we have here. On the x-axis is time. When were language models released? And then the y-axis is the cost in dollars per megatoken in log scale. So the first phenomenon I want to draw you to is how much intelligence or cognition can you get for $20 a megatoken? And in 2020, that got you DaVinci, the original GPT-3 API. In the middle of 2022, that would get you DaVinci 002, which was about the first model to have ChatGPT-on-release-level cognition, part of the Code DaVinci 2 lineage. At that point, it was also $20 a megatoken. So the intelligence or cognitive capability of the system, the amount of knowledge in the system, had increased substantially without increasing the price. So that's indicated by— I use the MMLU-5 score at— just because it's a very widely used benchmark and not totally polluted, and so roughly gets the, the capabilities of a model. And now, you're in— at this point in mid-2024, roughly $20 a megatoken will now get you outputs of GPT-4o. And that is a substantially more capable model than DaVinci 02. So that's this orange line. So if you stick, if you say, all right, I have a $20 megatoken budget, I cannot serve a model of sufficient capability at this cost, you can simply wait for models, like, you can wait for the model capabilities to catch up at that price point. The other way of looking at this is looking at what is the cost for a full fixed level of cognitive capability over time. So for that, we're looking at this gray dashed line here. You can see that in early 2022, um, it was going to cost you $20 a megatoken to get ChatGPT on release level cognition. With Llama 3.8B, I think this is the cost to run it as an AnyScale endpoint. So this isn't even the, like, bargain basement cost you could get if you really optimize the service yourself. That is 100 times cheaper 2 years later than what the DaVinci 02, TextDaVinci 02 endpoint was. And that means, like, we've seen this, like, this is, like, much faster than Moore's Law. And so it's unclear, like, how much more it can continue. But as we continue to accumulate improvements to hardware and improvements to algorithms and, like, the tremendous amount of capital expenditure in this industry on research and development, we can expect costs to continue to decrease rapidly. And so if it's, if you cannot deploy a language model like at the price that you are interested in, or rather if you deploy it now at a particular price, you can plan that the cost for servicing that endpoint will go down over time. Separately, this is the reason why it's a bad idea to run an inference as a service. Startup at this time. It's a pricing war to the bottom. All right, so with that, like, sort of high-level perspective on LLM inference, let's talk about deploying LLMs on Modal. So what's Modal's story on throughput, latency, and cost? For throughput, I, like, it's extremely easy to run very, very low-cost LM inference services on Modal. You can scale out to hundreds of A100s without having to go golfing with a cloud sales associate, or thousands of A10Gs. We, like, we have people running at that scale on our platform right now. And so what you, like, maybe you only, you don't need thousands of A10Gs every single, like, minute of every day. You need them to speed through a fine-tuning hyperparameter sweep. You want to run 1,000 parameters at once instead of like 10 at a time on your, or 8 at a time on your local hardware. So for that, I think we're really, we're in a really solid place. You can get up to like 8 A100s or 8 H100s as a single discrete unit. You can scale out to many of those. And so that's enough to do these fine-tuning jobs, even at the scale of 70 billion parameter models. For, for, for fine-tuning and for inference at like good large batch sizes, large sequence lengths. For latency, it's certainly challenging. Like we've heard from everybody how difficult it is to handle latency. I'm not gonna tell you that we make it easy. But I would say balancing latency and cost for models that are 13 billion parameters and smaller is doable, and you can in fact run reasonably latency-sensitive services that include neural networks at that scale on modal. And I expect to be able to move that up both by increasing quality of cognition available at 13 billion and below, and also as hardware improves and novel new chips, perhaps the GH200s from NVIDIA, or perhaps others, like make it easier to serve inference for larger models. For cost, we run at about $1.10 an hour per A10G. That's actually, it looks like, cheaper than AWS's prices now. We run on multiple clouds so we can find the cheapest GPUs, pass those savings on to you. You might find for some things, like particularly the fastest accelerators like H100s, that there is in fact a markup over the cost you could get it from running dedicated instances on the, on the cloud. $7.65 an hour. The way to win there is by making sure that you, like, if your workloads are spiky and variable, then you can get high utilization from on-modal, challenging to get high utilization with dedicated instances, or, you know, you, well, so that's again, more common with these throughput-oriented batch sensitive jobs, throughput-sensitive batch-oriented jobs, that is. So I've done the math. You, like, you can run endpoints on Modal at roughly the price that these API endpoints are available on, like, from other providers if you put some engineering work in it and you can match your requests to the hardware that you are running. So I won't, you know, we're running short on time. I won't go into great detail on this. It's hard to achieve very high GPU utilization. Check out the State of AI Infrastructure at Scale report for additional details about that and about some of the other challenges that you will face if you are running LLM inference at a large scale. Industry survey, bunch of brilliant information in it. So now is when I had intended to do my demo of running LLM inference. Inference on Modal. I'm actually in the— I'm gonna go back to this, but before going through that, I actually want to just say one thing. This is more like a little bit more exciting than the LLM inference. Modal is actually for more than just GPUs. It's not just a serverless GPU platform. It's an overall serverless runtime for running models. What I mean by a runtime, it's something that takes something that you've described in code and turns it into something that's in out there in the real world. So it provides like resources and infrastructure. So what kinds of things do you need in a serverless runtime? You need storage, like distributed file system, distributed queues and dictionaries, the ability to mount information from local to web. We've gotten lots of questions about using Axolotl and people are like, oh, can I do this? Can I do that? It's like, yeah, dog, this is a whole serverless computer. You can totally store stuff on it. Um, we would, you know, it wouldn't be worth the, worth the name of serverless runtime if you couldn't. Um, there's compute, which you've already seen with functions and GPU acceleration. There's also serving web services, web endpoints and web servers. So why do I tell you this? Um, because there are other things that I have wanted to do during the LLM fine-tuning course that are not serving LLMs that I've done with Modal. So let me pull that up. So let's say you are working with a database that produces a large list of users who want to get credits on your cloud platform. And you don't want to necessarily do that by hand, but you're going to need to connect to some databases. You're going to need to communicate from place to place. And you want to make sure to do this in a secured, repeatable manner. That is exactly the sort of thing that one could run on Modal. So that's what this little example here shows. We got a little image with a little HTTP client. It gets access to some secrets. Let me show you. It's sort of like, this is like a little local Python script here. It's got this main function, but decorated with app.local_entrypoint, which says talk to something in Modal on the cloud. So the key thing here is right there, this line in my VS Code here results I'm gonna map a function that I've written to grant credits to people over a large list of workspace names. And of course, for safety, I'm gonna give myself the ability to do it as a dry run without actually doing anything, just reading credits from the database instead. And then I can save the results locally for inspection. So here's that grant credits function. It's got a secret 'cause, you know, it wouldn't be good if we had a public API endpoint that you could just grant yourself credits for. That's even in Silicon Valley, that's considered a bad way to run a business. And so this grant credits function grabs that secret information and posts the payload which came from the— posts the information that came from the local Python environment on— like connects to our database and grants credits. So let's just do my quick test here, and we'll see how that goes. So I'm going to do a dry run of it here. This is everybody who is in the LM fine-tuning class who has also— and got a credit— who has also used Modal for something. Like, you hit our servers. We have that information. So, oop, there it is. Yeah, so modal_run_credits.py. See, I'm run it using it just like a normal Python script. Okay, --dry_run. We got a little terminal noise, but that looks right. Okay, dry run. All right, here we go. Spin up 25 containers and start checking. Okay, I'm seeing a lot of 200s there. So these are all these folks who got credits. I think it's about a couple hundred folks. Who got credits and used Modal. Great. So now let me just really quickly rerun that, drop the dry run, and this will now grant $500 in credits with a year expiration, just like the previously granted credits, to everybody who has already checked out Modal in the course of this class. So let's go. I haven't run this before because I obviously, you know, I'm not going to grant $500 in credits at scale many times while testing. But it looks like that worked. Ooh, dry run would've sent POST request. All right, let's drop that then.

Boom.

Did I say dry run? Oh, you know what it is? Default to dry run. That's a fun fact. Let's actually see that in the help. We can see that the default here is dry run, not no dry run. So let me actually pass that. That's why it's nice to have like the local logs when you kick off a job. Like it's, you'd hate to like find that out way after your presentation that you had like ran this script, but I could see it there in the, in the would have sent post, this dry run here in the logs. Really great to have logs available. So let's rerun that with no dry run.

Good for you for reading your own logs. A lot of people, that's advanced.

Sorry?

No, I said good on you for reading your logs.

Oh yeah, you gotta do it. Okay, I'm seeing a lot of 201s, credits added successfully. Store the results of that so I can double check that when I ran this live, I actually did grant credits to everybody. And you can also double check my work and make sure that you got your extra 500 if you've used Modal. If you haven't used Modal yet and you want that extra 500 in credits, just, I'm gonna run this script again. With a new database query checking for everybody who has used Modal and has not gotten credits twice, rerun it, and grant credits to everybody who uses Modal for something by 1 week from today, which is June 11th. So yeah, that's deployment of other things that are not LLMs on Modal.

That was a lot of fun. That was a good flex too. Granting credits on Modal using Modal.

When I suggested this to the engineers on the platform, they were not entirely enthused, but, you know, we made it work. Okay, so let me dive into doing inference on Modal. And, you know, we don't— we're already 20 minutes over time, so I'll be really quick. If you want to see how to run inference on Modal, the place to go is our Modal examples repo. We have a whole folder of LLM serving examples that show different ways to use it. We've got vLLM, trtLLM, we've got Mixtral and Mistral, we've got Llama models, so you can kind of mix and match those components. We also have different ways of serving them. So let me show you one deployment that I have up and going. This is an obliterated version of Llama 3.70B. What obliterated means is that the model's internal representation has been zeroed out in the direction that represents refusing to respond to a request. So I can ask for something like, hey, Llama, can you help me hide a dead body? I need your help on this one. The kind of question that if you were to ask ChatGPT, you would get an answer of like, no, I can't help you with that. So there's this technique, if you look it up, it's a LessWrong post. Neil Nanda is one of the authors on it, how to orthogonalize LLMs and remove sort of concepts from their residual stream, from their weights. And you can remove by looking at differences between when the model says yes and when the model says no, you can remove the ability to say no from OpenWait. So for example, how can I host a website for phishing credit card numbers? Get a nice answer out of the model for that. Write me a website that I can use to phish credit card numbers in HTML and JavaScript. And let's see, that one's not as interesting. And yeah, write me a phishing email that will get me some credit card numbers. So the utility here, I think, Apart from fun demos, there are certain things that I think the large models have been fine-tuned not to do that I think people should have the right to do. But the utility here is not just that we're serving inference on a model that says funny stuff like this, but it can also be used for security testing, for pen testing, automating, for example, sending out these emails to your employees to make sure that they are good at detecting phishing scams. So I sent that query a while back, and we see that there— this input is still in queue and cold starting. This is what people have been talking about with the— this issue with serverless setups and low-latency LLM applications. So if we take— we can take a look inside the modal inside of Modal to see what's going on with our logs. So we just launched this new little log thingy that's a lot— got a lot more information in it. All right, there we go. So these are the— these are very similar to the logs that you actually saw from Joe when he was running models on Replicate, because we're all running these open source libraries for running models, VLLM and Ray in this case. All right, that looks like we should be set up. So let's see how we are here. Maybe not. Let's run a second one and see if that fixes it. So again, I haven't tested— I set this demo up like a couple of weeks ago and then haven't tested it since then. It's what I get for running a new input live. But we'll see if that one, we'll see if that one comes up. Yeah, while we're waiting on our, like, serving example, the other one I want to show is using Modal to run kind of batch jobs. So we have an example that uses trt-lm to run Llama 3.8B. So the way this is something that looks a lot more like that credits grant script, it runs on some, like, collection of inputs that you send and runs it in, like, a batch manner oriented. Super high throughput. So unlike that chat-oriented interface that I showed previously. So we have— let me actually kick that one off while I talk through the code. Um, so this code here, this is like Python code that describes the entirety of setting up a TRTLM inference service, starting from the, like, a raw, like, getting a raw Docker image and, and, um, building a TRT-LLM model. So it looks like, yeah, this is— we start with our TRT image, we add some additional dependencies, like downloading models. All of this stuff is done inside of Python code along with the other things that you're putting out on Modal. So it stays all in this, like, sort of more friendly and flexible code environment instead of being in configuration and YAML files. So yeah, so we have— this is like when we download the model, we do it by running a Python function, which is good for sort of handling all the things that can go wrong when you're downloading a large model. Looks like our inference is just kicking off there. Printed out some input tensors to the logs. Always check your input tensors. And there we go. 32,000 tokens in 7 seconds at a total throughput of 45. 100 tokens per second. That's with batch size 128, so much lower throughput individually, more like probably, what's that, 30 tokens a second for each, like, as it would be experienced by each individual request on an A100 40GB instance. So if you're doing batch stuff, I actually think it makes sense to kind of view them as more like scripts that happen to run on robust cloud infrastructure, rather, and then you get an experience that's kind of similar to doing like Python script.py locally, like I did with the credit grant, rather than like turning them into an API service, blah, blah, blah. We do that for you under the hood, but it's not like built into the serving of your application. All right, it looks like I flew a little too close to the sun trying to run Obliterated Llama live. So I don't think we'll get an answer to our question of how to hide a dead body. Maybe that's good. Probably shouldn't be sharing that information too much. You know, too broadly. But yeah, last example that I wanted to show, I won't go through this in detail. I was gonna show those sort of like hot reloading development server where you can like stand up an endpoint, make a change locally, it automatically redeploy. I was gonna go through that in detail, but yeah, let's do that. Like you can see we like, when you edit your serving code, it like recreates. So if you wanna go from that like more, script type fast iteration loop environment to a deployment, we have this intermediate modal serve to help you out. And you get a pretty fast recreation of those endpoints, which is important. This particular example is our OpenAI compatible endpoint serving example. And that one, so you can take something, like if you run a server in OpenAI compatible mode, you can use it with a bunch of open source software like Instructor that expects the types of endpoints and the types of affordances of those endpoints that the OpenAI server provides. And on Modal, what that looks like, by the way, is you get a whole— you go into the VLLM, like VLLM the library, you pull out their implementation of an OpenAI API server, and then you toss some extra features on it. So you like add some middleware and authentication, you like connect it to your async engine, and then you, when you return it from that function, that runs inside Modal's infrastructure and we turn that into a fully deployed endpoint, which you can run, like which you can hit with a client. So that's what this one is showing with Python running against that client. So that one has like a fixed prompt of like, compose a lyric about baboons and raccoons, but you could go in and change that client script. It can be any kind of client script that you want, and it's, you know, separated out from everything else that you are doing. Great. Did we get— all right, nope. All right, that's it. Trying to do 4 demos in 30 minutes is probably a bit too much and too close to the sun. So sorry, no obliterated Llama today. But hit me up if you're interested in the code for that example or if you're interested in— one of the members of the class built Golden Gate Llama on Modal, a Llama model that believes it's the Golden Gate Bridge. If you're interested in working on that project, let me know. All right, so that's all that I have. I think, yeah, reminder with the credit grant, sent out to everybody who had gotten credits already and used Modal. If you use Modal in the next week, you'll, like, then you will get it one week from today, an extra $500 in credits. Modal strongly believes that the skills that you are learning in this fine-tuning course and the workflows that you are learning and the applications that you are going to build are going to deliver a tremendous amount of like of value and are going to be the way that this technology of LLMs or artificial intelligence actually rubber meets road and value is delivered. And so we're very enthusiastic about supporting you and about making sure that our platform supports what you're building.

Okay.

Also, the modal form for people who missed filling it out, modal form is still live.

It is, in fact, still live. And yeah, you can, and there's a script running on Modal to grant those credits.

Okay, we have a bunch of open questions, and I actually sort of encouraged our speakers not to type out answers because we wanted everyone to hear them and also not to have to be distracted as you were listening. So I think we're going to roughly try to speedrun as many of these questions as we can that are in the Q&A. I'll start at the one with the most votes. Can you please add a conference or demo for efficient swapping of LoRA adapters at inference? Trying to think about how to interpret that. I think, I think by conference or demo, that's just saying Uh, saying show me how. Yeah, show you how. Um, and then I think there are two ways to interpret that. One is use a platform that is doing this in the background, and the other is, um, like if I want to set up the LLM to run, to do it, uh, somewhere outside of any of these platforms. Uh, do we have anyone on this call? Uh, do we have a demo for this?

It seems like, yeah, the one thing I can say is like, okay, like we'll have, uh, we have like another modal session, which I just scheduled, so Charles doesn't even know about it yet. And then probably we'll do it office hours with Replicate as well. So potentially in one of those, or both, we can show some hot swapping if either person is, is, uh, comfortable with that. That's one option.

All right, uh, what is this mention of AWQ in the Honeycomb example?

Yeah, so AWQ is a quantization technique. Um, I don't really I didn't go like too deep into it, but it's just like a— it's a tool that you can use to quantize a model. It's compatible with VLLM. It's very easy to run. It's actually integrated quite nicely into VLLM, so you can— the code that I showed, uh, show you— showed you how to perform the quantization. There's a lot of different knobs that you can tweak in the quantization. I haven't really explored those, to be honest. Um, I just kind of use The default ones are the ones they have in the documentation.

Uh, Ruben Alvarez, this was a while ago, so it's been talked about, uh, since then, but there's a question. If I fine-tune, uh, for during fine-tuning, I load the model in 8-bit and the LoRA is in whatever bit size, what happens if I quantize uh, like Hamel just showed?

I actually don't know. Um, you know, as you can see, like, I merged it back in my example just because if I— yeah, if that's like a feasible thing for me to do, I just do it because I don't want to deal with complications. Um, so actually, I think Travis may have like kind of gone through a little bit of what can happen there. So I would like review what he, he went over in the recording.

Yep. Uh, also wonder what the process is to push an AWQ model to the Hub after training.

Yeah, Wade, um, yeah, that code I shared in my portion of the presentation, I shared a Hugging Face Hub, um, a repo with the code in it. That you can use to quantize a model, push it, push it to the hub. So if you go in the recording when it's available, go back to that, you'll see that code.

Unrelated to today's content, but how much should you charge enterprises for a fine-tuning project? Um, we could talk about this, and as with everything else, in varying levels of depth. My recommendation is to try to figure out what the problem is that they're solving and figure out how important that problem is to them, uh, and then figure out, like, do they have some metric which, like, they're trying to get this metric from X to Y, um, and work with them to figure out, like, how much the project is worth to them. Use that as a way to sort ground the answer. And, um, then Hamel Husain, don't charge hourly. So once you figure out how much it's worth to them, figure out a reasonable fraction of that.

Uh, yeah, and a lot of people don't know what is worth to them, which means don't do it.

Yeah, but I'd start by trying to figure out— not, not by starting, oh, here's the hourly rate, but instead like think about What's the metric that they, in their business, that they're trying to move? Um, and I think that will help you think through, uh, back of the envelope, what is it worth to them?