Why Fine Tuning is Dead w/Emmanuel Ameisen

Yeah, fine-tuning is dead. Long live fine-tuning. The idea of this talk came, I think, mostly from just like some fun tweets. I tend to believe that fine-tuning is less important than it once was, and Hamel challenged me to like actually defend that, that take. And so, so that's what I'll try to do here. All right, so who am I? So I'm Emmanuel, is my name. I've been doing ML for, uh, gosh, almost 10 years now. Uh, I was doing data science originally, then I worked in ML education actually, so trained models for demand prediction data science, helped people train models in ML education. I wrote a practical guide on how to train ML models. Um, then I worked, uh, as a staff manager at Stripe where I trained more models. And now I work at Anthropic where I fine-tune some models. Uh, and also, uh, currently I'm helping out with efforts to actually understand how these models work. Very important.

And just to plug you a little bit more, just think you're a little bit humble, it's like you, uh, there's a website mlpower.com, you could like see Emmanuel's book. It's a classic in machine learning, and like I would say in applied machine learning. So definitely check it out.

I, I appreciate the, the plug. Yeah, check it out. I one day hope to update it with some LLM-specific tips. It's just general machine learning knowledge for, for now. And yeah, you can get the first chapter for free on that website, so No commitment if you hate it. This is like the most important slide of the talk. This talk is my opinion, not Anthropic's. Mainly I say this because Anthropic, like among other things, offers fine-tuning. So if they thought fine-tuning were dead, that wouldn't really make sense. And so this is mostly like kind of, yeah, like hot takes and beliefs based on just seeing the field evolve over my career rather than anything that Anthropic really believes. I've been training models for 10 years. I don't recommend it. This is really the talk in 2 slides. I think it sucks for a variety of reasons. If you've talked to enough people that do it a lot, as I'm sure a lot of you do, they'll tell you all of the horror stories that come with it. I want to talk about 3 things though. We'll talk about the horror stories in the third part actually, on the difficulty. But one, I wanted to see trends I've observed over the past, let's say, 10 years. Then some performance observations on the fine-tuning work that I've seen shared or various papers, and then we'll talk about the difficulty. First, trends. I think that in machine learning, the best way to have a lot of impact is to just be afraid of anything that sounds cool. Like anytime in my career when there's been anything that sounded like the cool thing to do, it tended to be the case that like if you did that, actually that was like vaporware and really you should be doing the boring stuff. And so what that means is like, you know, in like 2009 maybe like people were like, oh my God, machine learning is like a big applied thing now. We wanna train models. But really like if you looked at like delivering value, really what you needed is like data analysts and data scientists of like write good SQL queries. And so that's what you should spend your time on, even though it sounded less cool to people at the time. In many places, this is still true today. Fast forward to 2012, you know, like, ah, like the deep learning revolution. Maybe it's a bit early, 2012, 2014, let's say. You know, oh, everybody wanted to use deep learning. It's like, oh, you had like startups that were doing, you know, like, I don't know, like fraud prediction that we're using some random forest or like, ah, surely now we need to use deep learning. Actually, like, that was too early. At that time, it was very, very hard to get deep learning models to work. You should just use XGBoost. Do the boring thing. 2015, you know, it's like, ah, like now deep learning is in full swing. There's a bunch of papers, like the exponential is starting. Surely, like, what you want to do is invent a new loss function or like improve on like the theory of an optimizer. But really, like, if you want to actually make your model better in practice, you just want to like clean your dataset. Notice the obvious errors, fix them, and then that would get like about, you know, 10 times larger improvement in about a tenth of the effort. And I think in 2023, we have a similar thing in a way with like definitely training your own foundation models. In some cases, I think, and then also just fine-tuning. It's very appealing. It sounds very cool. As far as I can tell, it's actually rarely the like first thing you should reach for or the most useful thing you should, you should go for. So I think just based on priors, we should be suspicious of fine-tuning because it's the cool— it sounds like the coolest thing. And that like right away, like, it's the coolest thing, probably gonna be the worst use of my time. I made a little chart that I think illustrates this somewhat. Oh no, hold on, fine-tune an LLM. Yeah, okay, same thing, we talked about this already. I made a little chart. Um, this is— this beautifully drawn chart is like my take on sort of like if you're doing machine learning in a practical way, sort of like what was the way to, you know, best leverage your time. And so at the start, hopefully you can see my mouse, but at the start, you know, people just trained models, there was no fine-tuning because there were sort of like no models to take and to fine-tune, or at least it was like, you know, exceedingly rare. And so you're like, you trained your own like random forest or your own like SVM or your own whatever, like even your like MLP, and that was that. And then kind of with, not really when ImageNet came out, but like a little, a few years after with like VGG and then later on ResNets, you know, that's when sort of like fine-tuning I think became a thing that a lot more people started to pay attention to. You could take a pre-trained model on, you know, in these cases there were image models. You could take a pre-trained model and then like fine-tune it, uh, on your smaller dataset for cheaper and get something that was better than if you trained it from scratch. And so, you know, as time went on, that became like more useful. I think BERT was also a maybe like big moment where that started becoming useful for text as well, where you could like fine-tune BERT models or fine-tune other models. And so I would say that, you know, there's like this general trend that like, okay, like fewer people are training, more people are fine-tuning. And then, you know, around GPT-3, maybe a little after, because it took time for people to sort of like really pick up, you know, there was this sort of like concept of like, ah, do you even need to do like any backwards pass on your data at all? You could just maybe like take the model and maybe it just works, right? That's sort of like, I would say that concept is like the original promise of LLMs, right? Is that like, you actually don't need to train, they learn in context, you can just prompt them. And so I like this chart because it's sort of like, well, I don't know if fine-tuning is dead. I don't know if like it was just a blip or if like this chart will like kind of like go back up in prevalence, uh, but I at least like the sort of like trends of like, ah, you know, it used to be that really there was nothing else you could do than training, and/or at some point you could like replace your training with fine-tuning. And now there's this whole other category of applications where actually you don't need to do any training at all. Um, and so the question is like, ah, you know, how do you extrapolate that trend? And I think like the realistic, not fun, not hot take answer is nobody knows. But my hot take is that, you know, line goes up. And so I think we're going to keep having that orange line increase. Um, in the future. Maybe I'll pause like really briefly in case there's questions because after that—

To state the semi-obvious.

Yeah.

If we go back to your slides, you're like what not to do.

Yes.

Most things that you said not to do, they were the thing to do a few years later. So you're like, oh, don't train ML models. And then a few years later you're like, yes, you should be using XGBoost. And then you're saying don't do deep learning. But then I think the thing after this is like, You're saying like by 20, at some point later you say, yeah, you should do that. Does that mean that if you say not to train, that you shouldn't fine-tune now, that it's gonna be the hot thing and it's gonna be worthwhile in a few years?

I mean, I think like that, I don't know, right? Maybe. I think this is not true for all of them. Like notably, like I think invent a new loss function is still like not a thing that you should do, you know, even like almost 10 years later or something. So I think it depends. I think it's certainly the case that like, as soon as something comes up, like deep learning, people want to do it. And sometimes it will actually be useful. Sometimes it'll not be. And it's hard when it first comes out to know whether it'll be like the invent a new loss function category, or the use deep learning.

Let me, let me ask a couple of questions from the chat.

Yes.

We got one from some Simon Willison guy. Uh, I get the argument for not fine-tuning LLMs, but how about embedding models? Is there a relatively easy value to be had from, for example, fine-tuning an embedding model on a few thousand blog articles to get better quality semantic search?

I think that's interesting. I, I feel like to me that's a similar question to fine-tuning a model. Like, I feel like if you If you buy that these models are getting better, um, and, and that we're gonna like— I think right now we're focused a lot on improving the LLMs rather than the embedding models. Like comparatively, there's not that much activity in the like embedding provider space. But if you buy it, they're going to be better. Um, you know, I, I feel like you'll just have like very general embeddings that work well. I think where this gets tricky and where you might like always need fine-tuning or need like a different paradigm entirely is like you know, your company has like a product that's like the XS23 and like nobody knows about it like outside of your company or something, and you want to build like search based only on embeddings with this, I think like that might require either like some fine-tuning or some like embedding, some like combined, um, RAG, like which honestly is like what I've seen work really well, right? Where you do like a combined sort of like some keyword search, you know, like and, and some embedding search.

What about the, what about the case where, like, okay, with the, with RAG and retrieval, in the domain-specific case, like, a lot of times what people think is a good sort of ranking or retrieval can be very, like, specific, you know? It can be hard to, like, capture that in embedding, no matter how good the embedding is. Um, So do you think— yeah, like, that's the part that I— yeah, that's the part I wonder about the most.

Yeah. I think, like, you know, not to, like, add spoilers or anything, but I think at the end of the talk, I have a slide where, like, I think that fine-tuning is dead is, like, the hot take version. I think that, like, the realistic, like, thing that I believe could happen is, like, fine-tuning is, you know, 5% of the AI R versus versus like 50 or something. Um, and so I think it's totally possible that you can imagine that like, yeah, like for your very specific search where— well, I think it's complicated because like this is kind of getting into what I talk about later, but as these models get better, you can imagine them just being able to like in context understand like what your specific search is. And so like, you know, you could have like your LLM drive your search and like do some sort of like basically like query expansion, uh, just because it like really understands well the context. For like pure embeddings, I don't know. Yeah, it's possible that you always like would need to like fine-tune some embeddings for some, for some retrieval.

Is, is there any benchmarks or data comparisons, the Dmitry SS, that compare how good your results are from doing a better job of prompting versus, uh, fine-tuning?

Yeah, there are some. I have some later in the talk. I don't have actually like I have like RAG versus fine-tuning, which is kind of similar. I would imagine like prompting to sort of like be a little worse maybe than RAG. Actually, that depends. Maybe like comparable to RAG. So I have some, some, I looked up some of the papers I could find that I'll share after in the next section.

Great.

Cool. Yeah, so I'll say that like I'm also all ears if people here have like papers that are comparing these, this performance. I was surprised to like not find that much. I'm going to be completely honest, I didn't spend super long doing literature review, but I spent like 15-30 minutes looking for a bunch of papers I could find on this that I was, in addition to ones that I was already aware of, and I didn't find that much that I found was super informative. So this is an example, I think, from the OpenAI forum of like fine-tuning GPT-3 that is one of the first examples when you look at fine-tuning versus RAG, at least that I could find. I think is relatively illustrative of what happens in some cases, not all of them, obviously. But in this one you have the base model and this one, the fine-tune, I think is like— I added a circle because I think it's the worst-case scenario or something because it doesn't seem to be doing any better. Then you have context injection, which I think is basically RAG, if I remember well from reading the post. And then various different models. And so it's the case that sometimes fine-tuning doesn't work.

So I have a question about this that maybe you can help me understand. So I always see these kind of things about fine-tuning versus RAG. And then I get really confused because my fine-tuning always includes RAG, includes examples of RAG. And I'm like, what do you mean fine-tuning versus RAG? It's not a versus thing. It's like you do both.

Yeah, agreed. Agreed. I have— can I just say, like, I'll answer this in like 2 slides.

Yeah. Okay.

Yeah. Yeah. Agreed with you though. Um, well, okay, maybe actually the one thing I'll answer. So in 2 slides I have like a comparison, like including fine-tuning, RAG, and both. Um, but the other thing I'll say is like, I think that this is also a matter— like one of the reasons why this is a hot take I have is that I think it's also a matter of prioritization. Like, you know, if you're like at this point in your life or something and you have the choice between fine-tuning and RAG, I think it's very important to know, okay, which one's going to be harder and which one's going to give me the biggest lift. And then of course you can always do both, right? But still, if there's two options, you kind of want to know which one is the most efficient anyways.

Oh, yeah, that makes sense. I definitely agree with that too. You should do RAG first.

Yeah. Yeah. I feel like this— yeah, anyways, I would bet that at the end of this talk, we're like, actually, we run those things. But it's— you gotta, you gotta have a few outtakes. This is okay. So I think this is— yeah, exactly. Okay. So this is kind of— I said in two slides, but this is basically what you were asking me about. This was a paper. I link it here, kind of like comparing fine-tuning and RAG. It's like on relatively old models. The paper after has like more recent models. But as far as I can tell, that trend like actually like held. And this is kind of hard to read, but basically you have like, this is your baseline. You haven't done anything. This is you just do RAG. This is you just do fine-tuning. And this is you do fine-tuning plus RAG. And then I would like just ignore the sort of like, you know, this is like, do you do like, like some LoRA or do you like some like full fine-tuning? And then I think this is like, I don't recall, these are like different prompting methodologies if I remember well, but anyway, or yeah, this is like the, the like fine-tuning dataset that you use. Like, is it formatted as a question answering pair? Anyways, you find that like if you look at all of these models, really, the increase comes mostly from RAG. Notably, this is even more true for the larger models where you get 58 with RAG and with fine-tuning plus RAG you get 61, and with fine-tuning alone you get way less. This is less true for the small model, right, where you get quite a bit more with fine-tuning plus RAG. But if you look at that trend, especially for base and large, basically you've gone you've gotten almost all of your benefits from RAG. And it is technically true to say, ah, if you do fine-tuning, you'll get more benefits, but you'll go from 63.13 to 63.29, as opposed to a 10x— well, going from 6.7 to 63.

So I think it's worth stopping here because I think this is actually a very confusing— people get stuck on this. I'm of the mindset, hey, if your model needs context from your data, you should always do RAG. You don't want to try to fine-tune all of that knowledge from all your documents and everything, like try to expect that your model is going to memorize all of that and stuff. I just don't think that's a good idea. So I feel like, yeah, I feel like if your application could use RAG, you should do RAG. There's no— yeah, I think people get confused, like, when they see papers like this, like, oh, fine-tuning versus Bragg, like, oh, maybe, you know, they're like, totally, like, no, there's no option. You have to, you have to use Rec. Well, I mean, like, in most applied situations, like, to make it work.

I think this is sort of like, you know, this is why, like, you doing this course is good though, because I don't think this is common knowledge. Like, in particular, the, like, you know, like, maybe you other practitioners that I've talked to, like, some people know, or know, or have the, like, intuition which I think is correct, that like fine-tuning isn't really the right solution if you want to like add knowledge. Like, it's like, it— that's just not what it's for in most cases. Um, and so like, you know, like, yeah, you can like say like, ah, for this use case actually it makes a little sense, for that one maybe a little bit more. But I actually think that that's not well known. Uh, and so like one of the, one of the reasons that I'm like on this hobby horse or something is to be like, no, like in most cases, like you're like, ah, like my problem is that, you know, like this model doesn't know about whatever our business model is. And it's like, yeah, the solution for that is not fine-tuning usually. It's like, just tell it what your business model is. So yeah, makes sense.

Yeah.

Um, I think I have— yeah, this is similar. I found this paper that was doing some like, yeah, again, like RAG plus fine-tuning. Not to like belabor the point, but like, I think I was curious on like, like one thing that I was curious about is like, oh, okay, like I think there's probably like a model size thing going on. Like anecdotally, like I think there's like some papers and some like, you know, like various experiments that have been running where it seems like fine-tuning is like more beneficial on like smaller models than bigger ones potentially. And so I thought it was interesting to find out this paper was like doing this with small models, small-ish models. And I think like this is another example of what we were talking about, right? Where like, I don't remember what the use case is for this, but oh, it's like, it's like for knowledge. And so it's like, yeah, for knowledge, even for small models, like you, you know, you want RAG.

And so how do you interpret this table? Like the ones on the very right-hand side, the FT-RAAG plus RAG, does that mean it's getting worse with fine-tuning and RAG?

That's how I interpret it.

I think a little bit surprising.

Yeah, my interpretation is that this is within like the noise. Like I would, I would like guess like just based on like base plus fine-tune being like pretty close, even like worse here, and like pretty close, that this is just like base model and fine-tune in this example, like, your fine-tune doesn't do much. And like, I wouldn't, I wouldn't over-index on this being like slightly lower. Basically, I'd say like, yeah, fine-tune plus RAG probably just does as well as like RAG would be my—

you know, it's interesting to look at this without reading the paper because we don't know what task is being scored, or at least I can't tell. And then it's like, if you wanted to measure adherence to a writing style book, then I suspect RAG doesn't do so, so much for you. Like if it's just writing style and fine-tuning, like I think, right, we could, we could pick a task and then tell any story, get the results to tell any story we want, but it's just a matter of what task are we optimizing for.

Totally. Okay, so I think this is a great point because as I was doing this and, you know, I was like, uh, writing my slides and like giving like a, a diabolical laugh, like, hahaha, like RAG is beating fine-tuning or whatever. I was like, well, okay, I should do a search for papers that show fine-tuning beating RAG. I didn't find many examples. I think a lot of the examples that I've seen are Twitter threads or something. Anyways, this is mostly a call for either the host of this workshop or anybody that's attending. If you have good papers that show this for what you're talking about, like style examples or things that fine-tuning is more suited for, please send them my way.

I think it's hard to explain because even when I try to explain knowledge, like, hey, like, hey, fine-tuning is not good for adding knowledge. Like, that word is not fine-grained enough. Like, what do you mean adding knowledge? Like, there's certain kind of knowledge that does make sense. Like, and they're like, oh, what kind of knowledge? And I'm like, oh, okay. Like, I get really, uh, it's, it, it's, it becomes an intuition, but like, I haven't expressed the intuition like completely clearly as much as I want to.

Well, and my like maybe maybe like scaling pill or whatever, like hot take, is that this, this intuition, or, or like the, the boundary between those changes with every model generation. And so it like, maybe like a good example is like, I think it used to be the case that like, ah, like you could say like learning like a style of speaking or something is something that, that requires fine— like, like some like style, not knowledge, like, like way of saying things requires fine-tuning. But like the better models, the like more recent ones can learn like a style from like a 2-line prompt, which the older models can't do. And so like for that, it's like less true, but there's still other things where like maybe it makes more sense. And so I think that like adds to the concept, like what is knowledge is, is something that changes with every, every model generation, basically.

Yeah, no, that makes sense. I, yeah, I have that same experience as well.

There are a bunch of cases where we could like look at it and we'd be like, we're not even sure if that counts as style or content. So if we were the main— fine-tuned on manufacturing copy from the makers of like the XS32 widget, and everywhere when this says like, the best widget is, and then you fill in the blank, it's always XS32. Like, that's sort of knowledge. That's knowledge that XS32 is some great widget. But actually it's just, well, that's whenever they express positive emotion, That's like the widget that they express it about. And so it's sort of tone and maybe knowledge is actually like not a very clear abstraction.

Yeah. I mean, notably, like, this is like way outside of the bounds of this, this presentation or something, but like, it's not clear from even like the early work that we have on like interpreting these models that, you know, like the other concept of knowledge as we're discussing it here, like is, is something separate from the concept of style within their actual weights, right? Like I would bet that. For many cases it isn't. And so it's not like there's like, ah, like, you know, that kind of like thing that's in, in like this, this like attention head or whatever, like, is the knowledge versus something else. Like, I think, I think it's, it's even the model doesn't have like that clean separation.

We've got our first audience, uh, question. You want to go ahead, Ian?

Yeah, sure. Uh, hi, thanks for this talk. So my company, we have a very complex knowledge base that we've curated, like 100,000 hours, I bet. Of time for precision oncology, which is genomics and cancer. My intuition is that I'm going to be able to curate using those curated rules, a fine-tuned model that does a good job at creating a first draft of a curation, right? So we curate what are called guidelines and clinical trial documents. Does that fit in your model? What, what would be your advice based on that description? Boy, I think the part that's hard, just going back on what Hamel was talking about, is that as a non-expert in this domain, I think I have a worse intuition than you do on like, where this falls on the like, knowledge versus style spectrum or something. I think one comparison I would draw here is like, maybe the Actually, I talk about it in the next slide, but there's like some attempts by, you know, people to train sort of like LLMs that are specific to finance or to agriculture or whatever. And those often, like, in the short term beat the current open models or current available models. But then I have an example after they like, they often are just beaten by like the next, you know, bigger, smarter model. So I think that's one thing I would consider is sort of like, what's the trend of maybe if you take like the Claude 3 models or like the GPT models and you take like the smartest and the dumbest one and like you see how they compare with no fine-tuning, that could give you a hunch of like, hmm, the next generation is probably actually going to be good enough. And then the other thing that I like to use to like decide whether fine-tuning will help or has a chance to help and also whether I need it at all is like to just keep adding a bunch of examples to my prompts of basically the shape of what I would fine-tune on. I feel like probably other people have said this in this conference or something, but seeing how well model performance increases with that also can give you a hunch of how well it would increase with fine-tuning on a large dataset. Notably, if you see it plateau after a while, you don't need to fine-tune, in my opinion.

We have a related question actually from— Oh, sorry. We have a related question from Simon Willison, who asks, does fine-tuning ever work for adding knowledge? I see this question come up all the time. And I think, like, the thing that's confusing about answering this question is, like, there's some knowledge that is, like, kind of intrinsic to, like, maybe, like, a world model or something. Like, I think about, like, okay, this could be wrong, but, like, a physics constant of, like, gravity of the Earth or whatever. Like it's like you— I think it's like for my intuition tells me language model is okay with memorizing that has seen that like so many times. I don't want to retrieve that with RAG but something more specific about Hamel Husain, like about my life, changing facts. Okay. That makes sense. Like something like, you know, that is not trained on clearly RAG, but there's some middle grounds of fuzzy middle ground where I have strong intuitions, but I don't know how does tell people? Like about adding knowledge, like what is, yeah, like I understand like intuitively, like there's some knowledge I want the language model to internally like grok, like fundamentals about the world or in things, but like others, like I would never expect it to like, you know, internalize. So I don't know, like how do you explain this?

Aspect. Yeah, I think it's also complicated, right? Because like, I think for most— so, so the first thing I'll say is like, for most things where I like, I want to add knowledge, let's say I want to add knowledge to like some model, you know, that I don't know, like, I like strawberries or whatever. So when it sees my name, it like knows that like, oh, by the way, like Emmanuel likes strawberries. I think that like, first of all, you know, that's almost always something that you could just do with like a prompt, right? Or like some RAG, right? It's like, oh, when there's like a question about me, like we retrieve something and it was like a description about Emmanuel, like, he likes strawberries. And if you have a good instruction following model, it'll just do the same thing. And so then the question is like, okay, like, if I was to like change the weights of the model for it to do this, like, how do I do that? And I think this is like often actually like not that trivial because if you just like fine-tune on a bunch of prompts that are like, what does Emmanuel like? He likes strawberries. If you have like a dumb model, it'll only tell you that I like strawberries if you ask it what I like. But oftentimes what you want with your fine-tuning is you want it to like know this information and use it when it's relevant in other contexts. And so maybe then you have to like Think about like, ah, like what I want to fine-tune is like, you know, I don't know, like we're a business that does like shopping recommendations. And so when like Emmanuel logs in, like, and he just asks random questions, I'm just gonna like be like, oh, like by the way, like did you buy your strawberries this week or something? And then like you fine-tune on like these specific prompts for this specific context. And then I guess my, my qualm with this and calling it like knowledge, like are you adding knowledge to the model or are you like basically fitting it so that like in this narrow distribution of these few prompts that you've given, it leans more towards mentioning strawberries. I think this gets fundamentally how these models learn. At least as far as I know, we don't actually have a satisfying answer to this. But I'm pretty convinced that a lot of fine-tuning ends up in that surface-level realm of, ah, in this specific context for this specific question shaped like the fine-tuning dataset, I will tell you this thing versus I've now learned, whatever that means for a model, that Emmanuel likes strawberries. Hopefully that made sense.

Yeah, no, I think there's no right or wrong answer. I don't think—

I think the answer to a lot of this stuff and the reason why it's so fun to work on is that we don't know, and hopefully find out soon. But yeah, I have a few examples. Oh yeah, go ahead.

Oh, did you— yeah, give a second for another question. Yeah. Great. So in terms of knowledge, okay, so one of the examples that I feel like perhaps has worked out that I've heard about is, so if you take like multilingual models and they, if those are not highly trained on non-English, but they have some non-English of the other types and then, so it sort of understands you know, a little bit of that language, but in general, its quality on the non-English languages is kind of crap. If you then fine-tune a lot more on the other language you want to get better quality on, that tends to work. And it seems like it's sort of like giving it new knowledge, but it's not like it already knew some of that language. It was just, it was just kind of crap at it because it didn't have a lot of examples. Is that maybe a case where that makes sense?

Yeah, I think that's interesting. The first thing that this brings to mind is that, like, in some of the interpretability work that we've shared, like, if you look at some of the way that the model represents, like, at least large competent models represent various things, like, kind of like the concept of a table, or, you know, like, whatever, like the Golden Gate Bridge, it'll represent it in the same way in different languages. And this is like increasingly true as the model gets smarter. Or something. Uh, and so I could buy that, like, fine-tuning that to be better is, like, slightly easier than other, other types of fine-tuning, because it's already seen this, it's already kind of, like, learned to map different languages to, like, a concept for other concepts. And so it, like, needs a relatively small change to, like, map, um, you know, this, like, new concept that it somehow, like, hasn't really learned fully for this new language. Um, to be the same, like basically part of the model that does the, does the, the reflection for it in English. So I don't know, like I could see that work. Um, right, for, for the specific thing— oh, sorry, go ahead.

No, no, and so similarly, like, does it— do you think that is the similar kind of thing that happens when people make, uh, you know, fine-tuned code models that are good at doing software, uh, you know, programming essentially?

Or would you think that's a different Yeah, I think with fine-tuning models, there's like a bunch of other concerns of like, it depends kind of what you want, but if you want like a, like code complete, this is like the style we were talking about, right? You kind of want the model to like, not, it's not going to be like, oh, hi, like I'm Claude, let me tell you about code. You just want it to like autocomplete the current line you're writing, um, which, which I think like that was like the style discussion we were talking about. And then there's like speed concerns as well with, with language, I think that like the knowledge of your codebase actually, like, that's also something that RAG works really well at. And I'm not sure you need like fine-tuning for, right? Like, just like, even having the good context of like what you're currently working on. Like, I'm not convinced, maybe to be clear, that like, I don't think there's good data on this. This is my hot take. But I'm not convinced like fine-tuning on your whole codebase or something like gives huge gains compared to like putting as much of that codebase in the context. The other thing I'll add that's related to this is that like, you know, both I think for like, Google's— some of one of Google's launches recently. And then like for the Claude 3 launches, there was like an example that was shared of like a model not knowing one of a rare language. And then you put, I think, like 100 pages of that language in the context. And then all of a sudden it can do it. So like without any fine tuning. And the reason I mentioned it is, is like, I think the fact that this is comparable to fine tuning is like really interesting. And I'll talk about it a bit more after, a bit more why I think it's very interesting. Okay, I'm going to jump ahead. Feel free to interrupt me if I haven't answered your question properly or if you have other ones. This is basically what Hamel Husain was saying. Fine-tuning is not the solution for domain knowledge. This was the other paper I mentioned, which is, I think this is an agriculture, yeah, agriculture paper. And you know, it's like, ah, if you fine-tune like GPT-4, you get 61% performance. If you— sorry, if you do that and you do RAG, and if you just do RAG, you get 60%. So again, like, you know, strictly, if you really care about that 1%, useful, but really you got most of it from RAG. And also, like, I don't know how big the error bars are here, but this is kind of confirming what we were saying, which is like, this is like a domain knowledge kind of thing. And so it feels less useful. Right. So the other thing that I think is challenging for fine-tuning, especially if it's like at the cutting edge is that you're aiming at a moving target. Uh, you know, like there's many labs, Anthropic among them, that are like continuously working on making them all better. And so this is, uh, this example is like the Bloomberg GPT example. I'm realizing here, I apologize, I think I've like used a bad figure, but essentially the Bloomberg GPT model claims that it was doing better than ChatGPT at the time, or GPT-3, on like some financial analysis task. And they like pre-trained their own model, so it's not fine-tuning, it's pre-training here. Which I guess doesn't show really in this table. But then a few, I think like 6 months later, GPT-4 came out and ChatGPT, and it was just way better than their fine-tuned model at basically everything.

So I have a question about this. That's really interesting. I'm glad you brought this up. First of all, the Bloomberg example, when they made a big deal out of it, like, hey, we did this pre-training of this model. It costs like millions of dollars, whatever. But my first reaction was like, why did you pre-train a model? And why didn't you fine-tune a model? But that's a different question. Second one is like, I'm curious, like, okay, yeah, these frontier models or whatever you want to call them, they're getting better. Like you're saying, it's a moving target. If you have a fine-tuning pipeline, let's say one for like Claude, just or just because I don't want to call attention to OpenAI, just to keep it, uh, so don't get kicked out. It's like, if you have a good fine-tuning pipeline where you have like, you're fine-tuning these like even big models, can't you just keep moving with the state of the art? Like, hey, okay, this new model came out, let me fine-tune that and let me keep fine-tuning that, assuming that those APIs are exposed. Um, yeah, like for the most powerful models, maybe they're not exposed yet, But just, yeah, curious.

I mean, like, I think like the question is, you know, like, can you always take the latest model and fine-tune on it? I think the answer is yes. Although like, obviously, if you, you know, like if you take the Bloomberg GPT example, they pre-trained, right? But like, presumably, the whole point of their exercise was like, that it's a large dataset that they have. So the cost to fine-tune isn't like much cheaper because they probably like mostly trained on their dataset. And so it's like, do you want to pay that much money anytime there's a new model that, um, that appears? Like, that can often be sort of like, ah, like you could either, you know, if you have like a pipeline that just takes an arbitrary model, like does some RAG and some prompting, you can swap models easily. But if you like have to like re-fine-tune, that gets pretty heavy. So I think it's possible, but it's just a matter of cost. And then the other thing, like on that note is, you know, like, I think that like, the fine tuning of like, a larger and larger model seems— and I'm curious, maybe you have like, more experience on this. I tried to find some papers, but that's mostly like an anecdote, seems like less and less effective or something? Or seems like these, as these models just like get better, better everything. And then, yeah.

Okay. So the, the kind of the most common tactic is to use the data generated by like the most powerful model to train like the one that's like, uh, like the faster models, like one step below it. Yeah. And keep, to keep walking that ladder up. Like, okay, new model came out. Okay, let's use that, get better data. And like whatever, and like, of course analyze the gap, but it's usually like, okay, you get a much faster model, uh, and hopefully similar performance or better performance, I guess. But like, yeah, you have to like look at the cost because there's probably like some— you have to do the analysis, like, does this even make sense, this exercise?

Yeah, exactly. I think that like, I think that a lot of— at least as far as I can tell, like a lot of teams, companies are like underestimating the cost and overestimating the value. Of this stuff. But yeah, I think you certainly can. Again, I'd love to, like, I wish there were like a few more examples of, you know, we were talking about where it's like, I haven't seen many papers where it's like, ah, we do this and we train like the super small model and it like actually works better. I think I've seen this like for some like evals that seem like obviously overfit to in some, in some examples where then like the model doesn't generalize. And so it's like nice for a paper, but can you actually use it for anything useful is not clear to me. But yeah, you can do it. I mean, I think it's a matter of cost at this point, right? And of like value. I think there's like certain use cases where it totally makes sense. I think there's certain use cases where it's like pretty far down the, down the priority list, in my opinion. I have— oh no, yeah, this is what we're talking about. Okay. So unless there's any questions, I think this is basically what we're talking about. And I'll just dive into the difficulty.

Yeah, please.

So This is something that like, I feel like Hamel, you've made like the same graph or like at least we've talked about this exactly. Or it's like really like for like when you're doing ML, you know, like the optimal use of your time, like even if you're doing fine tuning, to be clear, even if you're training models from scratch is usually like 80% data work, collect it, label it, enrich it, clean it, get more of it, see how it's broken, you know, 18%, let's say just like general engineering. Like how do you serve your model? How do you monitor your model? How do you make sure that like, works, there's no drift, all that sort of stuff. And then maybe like 2% debugging some like, my model doesn't train, or I get a GPU oomph, or something like that. And then you're like, 0% usually cool architecture research, at least that's been my experience. And so the reason I mentioned this is that machine learning is hard, I think, even if you don't train the models. If you actually buy this I'm not going to fine-tune, I'm just going to use RAG and prompt LLMs. You still need to do basically all of this. You filter out maybe like this, you don't have the model code, but you still need to set up input validation, set up filtering logic, set up output validation, monitor your inputs, monitor your latency, monitor your outputs, do backtesting of your prompts and RAG systems, do evaluation, have a train test split so you can do experimentation, potentially A/B test. There's this whole world of things. I think maybe The reasonable version of the hot take is not that fine-tuning is dead. It's that if you talk to me, I will only allow you to do fine-tuning if you've done all of this first or something. Because I think all of these are higher on the hierarchy of needs than fine-tuning. Which, by the way, I think is something that— you had this great article in O'Reilly recently that basically laid out the same thing. So I don't think it's a super controversial take. But I think the thing that grinds my gears with fine-tuning often is that people are like, don't do any of this and then fine-tune a model before they even have an eval, which is, I think, problematic.

Yeah, that drives me nuts too. Can you go back to the previous slide if you don't mind for one second?

Yes.

Okay, so this makes sense. One kind of question I have is like—

These are not very scientific numbers, if that's okay.

No, no, but it's fine. So the collecting dataset part, okay, you don't need to do that. With, uh, if you're not fine-tuning.

But you still need some of it for your eval.

What about the looking of data? You're still looking at data. Like, for me, looking at data takes up the 80%. Like, it's almost the same in a way, like the cleaning and the looking.

Yeah, it's like, oh, totally. Yeah, to be clear, like, maybe this is like not the way I should have, uh, formatted this, but this is mostly like, if you think that like when you're, when you're going to do fine-tuning, you're going to do like very different things, you're not. You're just going to spend most of your time looking at data. I think it's true in both. Although I think like the, as I mentioned, the failure mode is that people just like don't wanna do this and they're like, ah, instead I'm gonna like go on a side quest to like fine-tune. It's not all fine-tuners.

Oh, I see. Okay.

Okay.

So I mean, you're still doing this 80% no matter what.

Totally.

It's just like even more dangerous if you just like, just go straight into fine-tuning without— Yeah.

Yeah.

Like, like the thing you're more of a foot gun. Yeah, exactly. It's like the failure mode is like you just like don't wanna do this and you're like, ah, instead I'm gonna like do some fine-tuning and like some random dataset I'm not gonna look at. And like do that. And I've seen that go poorly.

Makes sense.

Right. So like, I think basically, like, most of your time should be spent on all of this. And like, once you have all of this, then I think it makes sense. And then basically, the way I think about it, right, it's like, this is all the stuff that's necessary to like actually have a working ML, just like, system. And so like, before you even train your first model, you should have like the infrastructure to like do the fine tuning. And so it's like, Once you've done all of this, and I think you can sort of like consider fine-tuning, but doing it before that is sort of unwise. That's basically the same thing. You know, I think like the first things I would do, I always recommend to like either my friends or like people I talk to that are building is like eval sets first, representative eval sets, large eval sets, eval sets that are easy to run. Spending days working on prompts, I think like I can now not even count the number of times that I, you know, told somebody like, well, have you thought hard about your prompt? They're like, oh yeah, yeah, I've worked so hard on it. And then, you know, like I look at it and it's like in 2 hours I get it from sort of like 30% to like 98% accuracy. And I think that's like, I am not a genius at this. It's just like actually spending the time, you know, making your prompts clear, following like, there's a bunch of really good prompting guides, so I'm not gonna talk about it much here. We can talk about it. In questions if people are curious.

Like, yeah, I think one observation here is like really interesting is like, okay, the first and the last bullet, those are like the same things that you should spend your time on, I feel like, in classic ML.

Yes.

Like, and so it's like really interesting, like not that much feels like it's changed in a way.

Like, totally, this is the same message we've been repeating for years.

Um, but yeah, there So like, not that much has changed, but there is like some narrative like, hey, there's this new profession, AI engineering, don't necessarily need to do ML or think about ML. And yeah, I'm curious, like, what you think about that?

Like, oh my, like, okay, so my take here is that, you know, it always was the case that like, you should spend your time on that and not on the sort of like, you know, like, like math part of things, let's say. And now it's just even clearer because like the math has been abstracted away from you for in many cases, in the form of an API, you know, if you use like API providers for models. And so like, there's like a strong, I think, temptation for people who are just like interested in interesting problems, a temptation that I have and understand of like, no, like, I want to like, get back and like do the like fun ML stuff. And I think if that's your reason for fine tuning, It's bad. But then to the point you were talking about, once you've done all of the actual work of like machine learning and then, you know, you've done all this and you like realize, ah, like the only way to get extra performance in this thing that's important is like fine-tune or like get lower price or latency or whatever, then that makes sense. But I think like basically it's like this is the same thing that it always was, but it's almost like the gravitational pull of the fun stuff is even stronger now that, you know, it's like, oh, what? Like I don't even like get to like see the Jupyter Notebook. I just like have an API call. That's no fun.

Yeah.

The, the last thing I'll say is like actually pretty important. I've left it in the last slide and, and like I think it's, it's just like looking at trends and basically extrapolating on them and either like deciding that like the trend line is going to continue or it's going to break. And I think again the real answer is like nobody knows, but if you just look at the trends of like model prices and context sizes, so this is like model price for like a roughly equivalent model, not even equivalent because models have gotten better, but this is like the price of like a Claude Haiku/GPT-3.5-ish level model. But like Claude Haiku today is like better than, you know, like certainly like the models in 2021. So it's, you know, actually like even cheaper than that. The price has gone from like, I think it's like $60 per megatoken in 2021 to like now, if I remember well, the blended price is something like half a dollar. And context size, you know, has gone from like, I think it was 2K at the start, or maybe 4K, and now 200K, 1 million, 1.5 million. I think I've heard 10 million. It's possible that like both of these trend lines stop. But I think it's important to consider like, what if they don't? One thing that's not pictured here is like latency, which has kind of like decreased in the same fashion. So like models are faster and faster. It's like, ah, Like if in 2025 or 2026, like you have a model where it has 100 million context, it has crazy latency and it's basically, you know, like let's say if it keeps going like even 10 times or 100 times cheaper, like you just, you don't fine-tune, you just throw everything in context and these models are amazing at learning from context. And if they get faster, like you just get your response immediately. Um, and so I think there's like a really interesting question of like, Obviously you can't extrapolate, you know, like any exponential or even like straight line forever. There's always points at which they stop. Uh, and so the question is like, depending on when this line of like price per intelligence basically plus latency, which is very important in my opinion, like stops, then I think it tells you like, ah, what use cases should you even consider for fine-tuning? It's like you consider the ones where it's like you're well outside of the like context window limit slash like chunking through that context at these speeds that will keep increasing will take too long for your application or something.

And then there's like the prefix caching that, uh, starting, starting to be done.

Yeah, exactly.

You know if Anthropic may offer that at some point? Okay.

Um, this is an Immanuel talk, not an Anthropic talk. Uh, but, but yeah, I, I think that like, I, I assume like all jokes aside that like things like prefix caching will be like, you know, kind of like a common thing certainly in a few years, right? Um, And if that's the case, and you can imagine that your fine-tuned dataset, like, is easy to format as a prefix most of the time, then yeah, that makes that equation even different. And so I think, like, honestly, I think this is why I had this chart last, because I think this is what started the debate. Because I think the thing that makes me the most, like, bullish about, like, oh, we won't really need to fine-tune models as much, is this chart. It's like just the direction in which we're going. That combined with the other beautiful chart I showed earlier of, like, prompting growing, I think is just a really interesting trend. And if it holds even for a year or two longer, I think that eliminates the need for fine-tuning for many, many applications.

Yeah, there's one question from Lorian. Can fine-tuning replace few-shot examples, especially when I need to save on tokens? I mean, I know the answer to that. But I think one correlated question is I talk with people like Harrison a lot, you know, LangChain, and I'm like, oh, like, what are, what are some interesting things people are doing? And he's telling me that a lot of people are doing like dynamic few-shot examples. It's like, think about RAG and like, think about like you have a database of few-shot examples and you're just like pulling the most relevant ones. And he's saying like that works really well. Do you find, do you see that? Like, where, you know, like the people you talk to, that— are they doing that? Is that common?

Yes, I've seen lots of examples of this. This is, this is common because like a lot of the times few-shot examples become unwieldy because you want them to be like evenly distributed. So if you have like a complex use case where like your model could do 10 things, where it's like you kind of want like one example of each of the 10 things and maybe like one example of the model like doing it one way and one example of the model doing it another way. And so it's doable, but you can just like quickly blow up your context window. Uh, and so like fetching relevant examples is something that like works really, really well and is like pretty common, uh, nowadays. I would say that like maybe in like my hierarchy of needs, you know, there's like all the stuff we talked about and then there's like really work on your prompt. I like tweeted something yesterday or something because I, I like helped the 10th person with the same loop, but it's like work on your prompt, find the examples that don't work, add them either as an example to your prompts or as like one that you can retrieve and add conditionally. And do this like 10 times. And then only after that, sort of like consider anything else. That tends to work really well.

Cool.

Yeah. Well, thanks for having me. Hopefully this was interesting to everyone.

It was very interesting. Yeah.

Cool. Awesome.

All right. Thank you.

See you, everyone.

See ya.