From Prompt to Model: Fine-tuning when you've already deployed LLMs in prod w/Kyle Corbitt

The topic is deploying fine-tuned models in production. The first commandment and the most important commandment of deploying fine-tuned models in production is you should not do it. Okay? All right, this is obviously not universally true or else I wouldn't be giving this talk, but I think it's a good default. So specifically, if you have an existing flow that is working for you and you figured out a pipeline with a prompted model, that, or, or if you like are able to figure out a pipeline with a prompted model, that probably should be the default. There's a lot of advantages there. A big one is your iteration speed is much faster. If you notice like, you know, something's wrong, you can play with it. You can update the prompt really quickly. And it just leads to a better experience versus fine-tuning. And which, you know, we, I think there's a lot of tools and OpenPipe is one of them, but, you know, we try and get the experience as good as possible, but really honestly, you can't really beat the experience of like, you just change the words and rerun it and you get different or better results. So the default should be, don't bother with fine-tuning unless one of like some very specific things are true. Okay? Okay. And so this is what I just covered. Start with prompting. If you want to get a little bit more fancy than just like raw dog typing out the instructions, very often doing something like, hey, coming up with 3 or 4 examples that are really high quality and throwing them in your prompt, or even a little bit faster than that. Maybe you have a bunch of examples that you know are good and you can sort of like use a RAG thing where you, you grab the examples that are most similar to the, you know, to the exact document you're, or whatever you're analyzing right now. These are all strategies I would reach for before I reach for fine-tuning. Okay. Now, all of that said, there are some very specific and very good reasons why you would go past that point and why you, you should actually do fine-tuning. And in my experience, there's, there's 3 dominant ones. So, um, the first reason is, is quality, right? Um, so you often find that there's only so far you can get with just prompting, uh, as far as guiding a model towards a specific outcome. And if how far you can get with that is not all the way to, to where you need to be to provide a good experience, that's a great reason to do fine-tuning. Another really good reason is because when you're doing fine-tuning, typically the— I guess you could say like the Pareto frontier of like how far you can push a given model or a given size model, like performance-wise, is much further out with fine-tuning than with prompting. And so what that means is you can move to a much smaller model for a given quality bar And the upshot of moving to a smaller model is that you can get responses much faster. So there are a number of use cases. And we have users like this where you're doing real-time— you're doing phone system work or even chat work where you just want to get a really fast response. Or the double whammy of you want to get a user's response back fast, but you've got this agentic flow where it's— you've got several different prompts you're running in sequence to figure out all the stuff that needs to happen and then to get the response back to the user. And of course, every one of those takes a latency hit. And so if you've got several of those running and, and the user's waiting, you know, you're gonna have a much better experience if those are going faster. And fine-tuning can, can get you to that spot. And then the final one, and this is actually the one we see pushing people the most often, is about cost. So in a lot of cases, there, there's a lot of use cases where GPT-4 can do a fantastic job and, you know, latency is not an issue. You're just using it for classification or whatever. But once you're doing this on hundreds of thousands or potentially millions of calls every day, it just gets way too expensive from a pure unit economics point of view. This is a very strong reason why people will go to fine-tuning because, again, you can move to those smaller model sizes and still get a really strong, a really good performance. Of course, the cost per token is much lower at those smaller model sizes. OK, so we've established that these are— that one of these things is true. And again, you should not fine-tune unless one of these things or some other reason that's a very good one is true. But let's say we've established this, right, that with prompting, either we can't hit quality or latency or cost. OK, so now let's go into the actual fine-tuning part and talk about what we're doing there. Actually, trick question, or, or, uh, trick statement. We're still not gonna go to fine-tuning. Okay? So even if you know that this is true, even if you know it's like, hey, I am not going to be able to deploy in production, um, because I know, I just know that it's gonna be too expensive when I do this at scale, or I just know that like I can't hit my quality bar. Um, you should still start by writing a prompt that gets you as close as you can. And I'm gonna explain why that's important, or I mean, you know, you can get away without doing this, but why I think this is, this is for most people, for most flows, the way you should do it. So the first thing is it gives you a baseline, right? It's like, okay, if I want to know if my fine-tuned model is worth it, if I should be investing time in this, like, I need to know in the alternative where it's just prompted, like, what I'm coming from and what I'm trying to improve on. And so that's really important. And then as we're getting later on and where we're talking about evals, we're talking about, like, actually the user experience you're providing, This gives you something to compare it to and just make sure, hey, is, is, is all the time and effort I'm investing in doing this like actually giving me a return? This next point is, I think, a little bit of a subtle one that people don't think enough about, but I think a really critical one. And that is trying to solve your problem with a prompted model can give you very good information about whether the task is possible. So this is, this is actually something we see very frequently. Where someone will come to us and they're trying to solve a problem with fine-tuning. Either they haven't tried or they haven't been able to get it working with prompting yet. We'll have a conversation with them and say, "Okay, explain to me, can you do this with prompting?" They'll say, "Oh no, it's because the model, we have this very weird schema that we're trying to get the model to output and it's just like prompting can't get it there." or some other reason why they think it doesn't do it. But then they'll usually— and so I'll say, okay, well, we need, we need data to fine-tune. And often they'll say, oh no, that's, that's no problem. We have a bunch of data, like we have data we can train it on. This is not a problem. What I have found in practice is that it is very often, unfortunately, I'd say like more often than not the case where even if you think you have labeled data that should work for this, the data you have, if it's not good enough to stick in a RAG pipeline or something like that and make it work with prompting, there's a very high chance there's actually not enough signal in that data to do what you want it to do. And there was actually a fantastic example that I think Dan gave in the first lecture here about a logistics company where, you know, you may remember if you watched that lecture, he was talking about they were trying to go from the description of an item to the estimated value of the item. And I think there was a good assumption there, like a hypothesis that, like, well, the description of the item should tell you enough about the item that, like, a model that understands a lot about the world should be able to guess how much it's worth. Um, but in practice, what they found is that it actually didn't. It didn't capture enough information. There was enough randomness or reasons why people weren't putting the right thing in the value box anyway. Um, there actually was not enough information in that training data. Whereas if you're going with a prompted flow and you're trying to get that working first, you can, you can find out very quickly, like, okay, I just don't have the data here. You know, the model is not able to figure this out. This point is not a universal rule. Like, it definitely is possible to train a model that, through fine-tuning, that can, that can, like, have great performance on, like, your proprietary dataset or whatever that there's, like, no way that a prompted model could do. So definitely not a hard and fast rule, but I found, like, You know, that's if, if, if you can go in this direction, it's going to make your life much better. So just kind of like a heuristic on that point is the— so, so if you do find that you're able to successfully write a prompt that is able to at least, you know, more often than not do a good job on your task, then there's like a 90%+ chance that you will be able through fine-tuning to improve that further on the metrics you care about. So whether that's latency, quality, or cost, there's like a very good chance you can get this working. Whereas on the other hand, if like there is no way to formulate your task in such a way that a prompted model could ever work and you're kind of just hoping that it can learn from the data, I mean, that still can work, but you're definitely playing in hard mode at this point. Like this is now a hardcore data science project. Like you have to figure out, is, you know, is there actually enough signal in this to solve the problem? And it just gets much more complicated. And in a lot of cases, it turns out that there actually isn't, that your data is not clean enough or well-labeled enough or whatever to learn what you want it to. And so there's a high chance of failure. So you want to be in this regime. You really want to be in the regime where you know it works with prompting, and then there's a very high chance that fine-tuning is going to be able to juice your returns and get you in a way better place. And you want to avoid being in this other regime. OK, so we've got a prompt, and now we're going to fine-tune. Okay, I guess this slide is something I just wanted to share quickly. The conceptual way I think about it, the way I encourage folks in your situation who are thinking about deploying some production to think about it, is like, you know, you're going with the prototype. You're basically, when you're iterating fast, when you're trying to figure out, is this even something that's like possible to do? Is this something that's providing value? Is this something that, you know, with our whole, whatever app or flow or whatever it is you're building, and you're trying to see if it's gonna work or not, and if it's gonna provide value, just do all that part with GPT-4, don't worry about fine-tuning. And then once you've got something that actually works, that scales, that you have a sense what people are using it for, that's the point where you're going to drop in fine-tuning. That's, that's just kind of like the general mental model that, again, not universal. There's, there's reasons and times when it makes sense to go straight to fine-tuning the model if you can't get prompting working. But in general, this, this is the flow I would recommend trying to make work by default. Okay, so we have, so let's say we now have a prompt deployed in production. We have, you know, people using it, people prototyping, playing with it. What's the next step? Well, you got to look at the actual data. And this is going to teach you so much, both on, on like how good a job your existing prompted model is doing. That's super important, but also like on the types of things that people are using your product or your model for. And that's actually the part that I find personally is the most useful part of reviewing data, is it just gives me a very, a much better sense of like, okay, what is, it just gives me a feel for like what my input distribution looks like. And without that information, like you can make assumptions about, you know, the types of tests you should write. The, um, the, the types of, like, even the types of models you think will work well or poorly based on, like, you know, just like how hard you think the task is, the types of things you think this is being used for, that can be very wrong. So you just gotta, like, look through it. Um, this is, um, there's, there's, there's no magic here. Uh, the right way to look through it varies a lot. Like, very often in whatever system you are writing, you have some sort of natural UI, whether it's a chat conversation interface, whether it's like some kind of classification system and you're putting documents in different things. So if you do have kind of like an existing UI, then that's probably the right place to look at it in. If not, if for whatever reason that there's not a good way, then there are tools out there you can use, and OpenPipe is one of them. But it'll just give you a nice formatted, okay, this is what the input looks like, this is what the output looks like. Let's click through, let's go through 20, 50 of these and just get a sense of what our data looks like. And you're wanting to look at both the input and the output. You want to get a sense of the outputs any good as well, of course, but honestly, like I said, I find a lot of the value here is, is just getting a really good sense of my input distribution. Okay, so we've looked at our data. We have a sense of what's going on. What's next? So, okay, so this next one is like, I need to explain what I mean by this. So you actually, once you've, once you've looked at your data, Once you've looked at this data, once you understand, okay, this data is good, now you have to like actually— that is the data you should use. And let me like— the failure case I see here is, um, there's a lot of, uh, there's, there's like a tendency where, um, sometimes people will look through their data and they'll be like, oh, I noticed this like particular class of data, uh, the, um, the, the model I'm using in production right now, it does a bad job on. And so what they'll do is they'll go in, they'll like drop that out of their data set before they fine-tune on it because they're like, well, I only want to fine-tune. And they'll, you know, sometimes people have like a fancier automated way of doing this where they say, okay, well, you know, I, I'm going to collect user thumbs ups and thumbs down on my prompt, and then the ones that get thumbs up, um, I'm going to go ahead and fine-tune on those, and the ones that get thumbs down, I won't. And, um, I'm not going to say that never works, like, that definitely can work. The failure mode that I see here that does happen though is if you're rejecting all the places where the model's doing a bad job, um, there's a real danger that, like, there is some correlation between the things it does bad on and a specific space of your inputs or something that you actually do care about. And if you don't fine-tune on that, then you're just going to do a bad job there. So the real solution here is figure out why the model is doing a bad job, figuring out where in your input space is it doing a bad job, and then do one of several things. You can manually relabel it. You can just spin up a relabeling UI. And there's several of them out there that are pretty good. And kind of like type out what it should be. You can like try and fix your instructions. Oftentimes I find that's the best fix is like as you're manually reviewing the data, you're like, oh, it's doing a bad job here. And then you play around with it and you fix your instructions and you can get it to a place where it's doing a much better job there. So that's very often the right way to do it. But the main thing is you don't want to just like drop those ones and just fine-tune on the ones that it did a good job on because like there's a very high chance that means you're just like chopping off big chunks of kind of like your input space. And then when you have your fine-tuned model that you're deploying in production, it's never seen this data before, and it's also just going to do a bad job in that area. Um, so, so that's— anyway, no, no shortcuts there. Now I'm going to contradict myself a tiny bit, which is if what you find is the model is mostly doing a good job, and like 10% of the time, but it's kind of like a random 10% of the time it like forgets an instruction or gets something wrong or something like that. What I have actually, but it's not like super correlated in like this one area, it always gets it wrong. It's more just like, well, sometimes GPT-4, it's non-deterministic. It messes stuff up once in a while. I have actually found that when that is the case, it actually doesn't really hurt to train on that data. And I, this is what I'm saying right now is like very controversial. And I'm sure that like, you know, like, I could have a great debate probably with people on this call right now about whether this is, how big of an issue this is. But I'm just saying from my own experience, what I found is you can very common— Basically, these models are quite smart and they're quite good at generalization as we're getting to these larger models. And I'm talking really, the advice I'm giving right now really is for, I'm talking about like LLMs, I'm talking like 4, 8 billion plus parameter models. I'm not talking about the really small ones. But for these big ones, I actually find that like the training process itself, to some extent, like does a form of like regularization or normalization, where even if the input data is not perfect, as long as it is like pretty good on average, you can actually very commonly see the model outperform even the, the model that was used to train it originally because of that sort of regularization effect. We see that very commonly on our platform. We have lots of users who are training their fine-tuned models directly on GPT-4 outputs, and then they'll actually run evaluations between GPT-4 and their fine-tuned model and consistently see, for a lot of tasks, that their fine-tuned model is doing better than GPT-4. Again, I think this comes down to that regularization effect where GPT-4 will just, 10% of the time, make a dumb error, and as part of the training process, It sees those, but it sees the 90% good ones, and it learns, you know, that, that basically the, the goodness. So anyway, this, all this to say, don't stress too much if there's like a few bad examples. I think in many cases you can get great results anyway, and it may not be worth the effort of like manually reviewing all whatever 5,000 or 10,000 examples you have. Okay, so. Let's say we've got, you know, we're pretty happy. We've reviewed our data. We're like, okay, you know, at least like 90% of these look good. This is basically doing what I want my model to do. Okay, so next, before we actually do our training, very important, you have to pull out a test set. Okay, and this is not news to anyone who, you know, has a background in machine learning. But I do actually see this as something that sometimes people don't do, or Or often I'll see people who have a test set, which is kind of like weirdly constructed. So they'll like pull out, like, I don't know, they'll come up with like 10 random examples because they saw these are the ones that Prompt did poorly on, or these are ones that like, for whatever reason, you know, a customer complained about this. So, so they have like a way of constructing a test set, um, that is not representative of the input data, I guess is the way I would put this. Um, and I think that's totally fine. Like, I think having a test set like that of like specific cases that, you know, are weird corner cases and you want to make sure it's doing well on. Like, that's, that's a, that's actually great. There's nothing wrong with that. The problem is if you are testing on that exclusively and you are not also having a test set, which is just like a basically like a random subsample of your inputs, then you can be like, you, you can, you can have, you can think you're doing well and really not be doing well. Like, you really want to have a test set, which is just like, hey, grab 5%, grab 10% of my data. At random, don't train on it, and use that as the input. So highly recommend doing that. This is just kind of standard stuff for machine learning, but again, something I see people that don't have a lot of experience fine-tuning maybe not realizing is really important. Okay, so now let's talk about the actual model that you should be fine-tuning. So one nice thing is if you've got— and I know that you've had, you know, like you've already had a chat with Wing from Axolotl. A really nice thing about Axolotl or Hugging Face's SFT Trainer or things like that, and just the Hugging Face Transformers ecosystem in general, is it's pretty easy if you get a fine-tuning pipeline set up for one model to throw in several different ones and run them side by side. As long as your dataset isn't huge, the cost is not prohibitive. We're talking maybe a few dollars in many cases per fine-tuning run. And so that's great because it makes it a lot easier to kind of like try different things. But kind of as like a rule of thumb or a place to start, so this is a chart I put together actually for a different presentation, but I think it's kind of representative of the ecosystem and some of the models that people are fine-tuning commonly. And so the sense here, so this is actually based on real data. The, the, what, what this is normalized to is the number, number of training examples it took to match a certain performance threshold. So basically, for the specific test that this is, I looked at like, okay, how good does GPT-4 do on this? And then for each of these other models, like, how many training examples did I have to add until it was able to match GPT-4's performance on this specific task? And the answer to that question is task dependent, but this is kind of like, this gives you an overview. And what I find is, you know, in general, if you've got like a few dozen examples, like you can very often get like Llama 3.7 billion to match GPT-4 on your example. And then as you start getting to smaller models, it does take more training data. It takes a wider input of training data. And again, this is very task dependent. There are some tasks where like it doesn't matter how much training you do, you're never going to reach GPT-4's performance. I actually find that that's definitely the exception for most things I see people running in production. I find that these fine-tuned models actually usually can pretty easily match GPT-4's performance if you have a good training set. The place where I actually see most people coming in when they're actually doing this in production, really the sweet spot, is this 7 billion, 8 billion parameter model. Things like Llama 3, 8 billion, Mistral, 7 billion. I see that as like a really nice sweet spot because the amount of training data you need to get good results out of that is not overwhelming. You know, if you have like 1,000 examples or so, you can typically get really good performance out of this. And, and at the same— and so if you're running in production, like we were talking about earlier anyway, like hopefully you do have like a few thousand examples that you've kind of gotten that have gone through GPT-4 anyway. So hopefully you're in a good spot. And the cost savings are really significant. So actually, this, I think, yeah, I didn't update this with the latest inference cost you can get online, but this should actually be lower. Like, you know, the cost you're seeing per token for like Ollama 3.8 billion is somewhere in the 15 or 20 cents per million tokens versus GPT-4, even GPT-4.0 is I think about 50 times that. So it's a huge savings you're getting there. And you can get really good performance in that, you know, 1,000, 5,000 example range. So that's where I often recommend people go. But again, these training runs, the nice thing is they're really cheap to do, especially if you're in that, like, you know, less 5,000 or less example thing. And so, so why, you know, you can try a 70 billion model and see if it does better. You can even go smaller. You can try like a Phi-3 and see how it does. And like, ultimately, like, it's a pretty easy test to run. But, but this is probably what, where, where my default would be for, for a lot of tasks, at least. Okay. So now we're going to talk about evaluations. Evaluations obviously are a huge subject all on their own, and I'm not going to go super deep into this. I think probably there's other segments of the course where you're going to be talking about this more, but I do think there's some, there's like an interesting framing here, which is, which is not one that I hear people talk about that much. And I think it's pretty important. Because in my mind there's actually two different kinds of evaluations, both of which are really important. Um, and so the first one are fast evaluations, and fast evaluations are ones that you can run like in your training loop, or like even if you're doing like prompt engineering, like as you're updating the prompt, these are evaluations you can just like update the prompt, run, and then immediately see, did it good? Did I do a good job or not? So these should be relatively fast to run. These should be relatively cheap to run. And not require a lot of outside input to get good results. Where I personally have found a really good sweet spot for this category, these fast evaluations, is using LLM as judge. That's kind of the default. That's where I would start is basically asking GPT-4 or asking a jury of GPT-4 and Claude-3 and maybe another model or something to say, okay, this is the task, this is the input I had, and then these are two different outputs from two different models. And there's some tricks here, like you have to randomize the order because there's a slight preference for the former one. And if you're using the same model as judge and as one of the entries, it has a preference for itself. So there's a little bit of subtlety here. I think there's good libraries out there that can help you do this that are built in. We also, on OpenPipe, this is sort of like the default evaluation we give people. But the main point is these things are cheap enough to run if you're running it against, say, 50 or 100 examples that you can just update your prompt and rerun it, or you can fine-tune a new model and rerun it and really quickly get a sense, OK, is this plausibly helping me or not? And I think that having something really fast like that that you can run quickly and get a sense, OK, am I on the right track or not, is super critical. Because otherwise, if you get to— and I'm going to talk about the other kind of evaluation in just a moment. But basically, if you've got these slower evaluations that require you to run in production, your feedback cycle is so slow that it's just a lot harder to, to make progress, um, and to get where you need to go. So I really recommend investing in fast evaluations. Um, you know, I think Element Judge, as Judge is right, is great if you're doing— there's also like for specific tasks, other evaluations that can make a lot of sense. Like if you're doing like a classification task or something, then you can just get a golden dataset and calculate an F1 score or something like that. Anyway, but the high-level point is find something that you can run fast and have a quick inner loop and can make— help you figure out you're in the right direction. Okay, now the other kind of evaluation, which is also super important, are like outer loop evaluations, slow evaluations, production evaluations, and you can call it different things. But the idea is that this is the one that is actually measuring the final result that you care about, right? And so this is something like if you're writing a chatbot, like a customer support chatbot, it's like, okay, what percentage of customers came out of this feeling like their problem was resolved, right? And so these evaluations, I don't think there's like, I mean, there definitely is not a one-size-fits-all. It really basically comes back to like, what is the outcome, you know, the business outcome or the product outcome that you're trying to drive and figuring out how can I measure that. And so you really, these are really critical as well because, you know, the fast evaluations, even if a call looks better in isolation, maybe it's, if like a specific model's output looks better in isolation, maybe it is not, maybe there's like, you know, like some other interaction with some other piece of the system that's like giving you a bad result or maybe like the LMS judge is like not perfectly accurate. And it's not actually measuring, or maybe it's like once you deploy to production, like you're quantizing it or something, and there's like some disconnect in like the actual way it's running. So there's all these reasons why the sort of fast evals we were doing before might not tell you, might not give you the full picture and might not be perfectly accurate. And so you really want to have that outer loop and make sure that you are going in the right direction. Okay, so I have a couple of examples here just from OpenAI with ChatGPT. Uh, you know, in their particular case, I think they measure how often do people regenerate, how often do people give like a thumbs down. Um, they also have like a more concrete, uh, flow, which, which they rarely put up, but I have seen it a few times where, um, you know, you ask a question, it'll just give you two responses side by side. It'll let you choose which one is better. Um, and I think again, obviously it's really dependent. Uh, if you're not writing a chatbot, then probably this is not the right form. But I do think it's important that you think about, for your specific problem, how are you actually going to measure that you're doing a good job and that it's driving the outcome that you actually care about. Okay, so we're almost, we're almost getting through this. We're to the 9th commandment out of 10. This one is really important. So hopefully, like in this previous one, you've written these, these slow evaluations. You're actually looking, you have some way of measuring at least somewhat objectively or repeatably, like how well you're doing. Well, once you've deployed your fine-tuned model, you really have to keep running those on a constant basis. Because if you don't, what you're going to see is something is going to change about the world. There's going to be some difference in the types of users you're getting or the things they're sending you, or there's just like so much that can change out there that if, if you're not continually measuring how well this is doing, like you're going to have and like sort of the, the term machine learning practitioners use for this is data drift, which can come from a lot of sources. But the main point is, like, over time, it's very likely that your model is not going to be as well adapted to the input as it should be. And like, one, so one, one interesting concrete example, in, in our case, we had a customer who was doing basically structured data extraction from, from, from these call logs, actually transcripts. And they noticed that they had this eval that they were running in sort of this slow loop, and they noticed that their responses were getting worse right around January of this year, so I guess 5 months ago. And so it wasn't huge. It was like a, you know, I don't remember exactly what the numbers were, but it went up from like a, you know, 1% error rate to like a 5% error rate where it was like hitting their things and wasn't working. And so, so they ended up looking into it and they shared with us what they found, which I thought was fantastic, which was the data extraction. There were, there were like a number of fields they were trying to extract from these call logs and one of them was like a specific date. So it was, it was people calling in about, it was call logs about people that like mortgages basically and trying to get the due date on their payment, whatever. So one of the things they were extracting was like, what is the specific date that like this, the next payment was due? And what they found was, because their model had been fine-tuned entirely on data from 2023, not in all cases, but in like 5% of cases, even though we were now in 2024, it would just kind of like, you know, it would get the date, like the month and day right, but then it would like put the year as 2023 in the extracted date instead of the year of 2024. Because it was just like in every single case of the training data, it was the year was always 2023. It didn't see any other examples. And it didn't do that every time. Like, it was smart enough in most cases to figure out, okay, this, you know, we should be pulling out the 2024. But anyway, that was starting to happen. So all they had to do was retrain a model with a few extra, you know, I don't remember how to put in, they put in like, you know, 10 or 12, 2024 examples. And that was plenty to clear it up. So anyway, just an example, you never know where this stuff is coming from, but you do have to keep measuring to see if things get worse. Um, and okay, and so finally, um, the last one is I, I frame these as commandments. Um, I tried to give disclaimers as I was going through, but I think it's really important to realize that the most important thing is that you understand your problem, that you understand your data, you understand what you're trying to solve, um, and then you figure out if, like, what the right way is to solve it. So I think the flow I described is really, really helpful. I think there are other ways to do it that can also be effective, But I do think that like this is a good place to start, especially if you haven't done this before and you want to just maximize the chances that you're able to do it successfully. So anyway, yeah, it's like Pirates of the Caribbean, right, where the pirate says like, you know, well, the pirate code's not really a code, right? It's a guideline. Anyway, same feeling.