Fine Tuning OpenAI Models - Best Practices

So briefly, what is fine-tuning? Fine-tuning is taking, training a model to follow a set of given input and output examples. So basically, when faced with some sort of input, if you don't like the way that the public model is responding by default, you can teach it to output responses in a particular format or whatnot when it sees that input in the future. So when do you want to use fine-tuning is a question that we get a lot. It's very good for following a specific format, processing the input following specific complex instructions. So if the base model just kind of isn't following some instructions that you're giving it, fine-tuning can help with that. It's also good for improving latency and reducing token usage. So your alternative here is like multi-shot examples take up a lot of space in the input prompt and a lot of, you know, additional cost and latency. If you can teach the model to output or behave in the way you want it without all those examples, you can save on both the latency, the processing time, as well as, you know, the amount of money you're sending to us. So those are some things that fine-tuning is good for. Some things that fine-tuning is not good for that I think people still try, but it doesn't work that well, is firstly teaching the model new knowledge. So again, we're really using fine-tuning mostly to do, to follow a given format or tone, or to do some cost savings, but not to kind of add things that it previously doesn't know. Reg, or if we're in our case, the, the custom models offering that we have are better options for that. It's also not great at performing multiple unrelated tasks. So the, the best use case for fine-tuning is really sort of the same task over and over again, and not a combination of tasks. If you have a combination of tasks, either do prompt engineering or just create one fine-tuned model for each one of those tasks. And then finally, kind of on the first one, similar theme to the teaching the model new knowledge, it's not helpful to use fine-tuning for including up-to-date content because A, you can't learn new knowledge, but B, you're, you know, you can't fine-tune and deploy that quickly to get up-to-date knowledge into the response. So Um, what do you mean by custom models in this?

In that, in this, uh, yeah, yeah, good question.

So we have, uh, kind of a range of fine-tuning offerings. There's the self-serve API where you can just go into our, our platform, sign up, upload your training files, and get a fine-tuned model out. Um, and then we have, uh, a custom models program, which is actually you're sort of partnering with us over multiple months, kind of multiple million-dollar engagements and we take a large corpus of your data and work with you to train, retrain the model at a sort of deeper level. And so the, you know, we're able to incorporate techniques and research and so forth there kind of on a case-by-case basis that the self-serve API just doesn't have. And so that's for kind of a select group of partners that we've started working with. That's the way that we've been able to get them models that, that understand new knowledge. So for instance, one case study that we have is, is Harvey, which is this legal company. We've trained on a bunch of case law that was not, you know, the base model was not very good at, and given them a model that performs much better on kind of the law case law-related tasks that they have.

That's a really interesting example. Does that, like, is GPT-4 not trained on case law?

GPT, you should look at the case study because they have some data there on the output, but the base model tends to hallucinate more cases. There was, I think, that news article of, you know, a lawyer who got called out basically for making up a fake citation. from ChatGPT, uh, or some other, uh, some other model. And, uh, uh, so they, for their legal product, uh, they obviously care a lot about, um, reducing hallucinations down to as close to zero as possible. And, uh, the custom model does a lot better for that.

Gotcha. Um, we actually have one actually related question from Simon Willison. Simon Willison is asking, are there really, are there any really good fully fleshed out examples of fine-tuning against OpenAI models with realistic, real-world examples of fine-tuning training data and examples of the kinds of prompts that can be answered using the fine-tuned model that don't work as well with the default models?

We have a good example in our cookbook of, um, a Q&A model, um, that is also— that is trying to teach the model to respond with "I don't know" more often if the, uh, question is sort of not in the theme or not included in the examples. And you can see from that, that you're kind of, when you measure how often does the model say I don't know when it actually doesn't know, that that's higher with the fine-tuned model than it is with the base model. So cookbooks.openai.com is where you can go to find some of those notebooks. It's the first one that comes to mind for me. Also just kind of wanted to call out on our developer site this new guide we have for optimizing LLMs for accuracy. There's the link there at the bottom, but we talk about kind of, you know, when to use fine-tuning, when to use RAG, when to use both, when to use neither. And in that guide, we have this chart here where basically the, this kind of going up on the chart means you're adding more context, more sort of new information to the model. And then going to the right on this chart is optimizing the model, or changing rather how it sort of responds and acts to your inputs. And we find that kind of a lot of use cases Our base models are really, really good. And so most people end up, you know, just in sort of this, this bottom left quadrant here, where all you need is, is prompt engineering, maybe few-shot examples. As you go to introduce more context, you know, teach the model more things, she'll start to move in kind of the RAG direction. If you're primarily looking to change the way the, the model follows instructions or formats responses, you know, you're moving in the fine-tuning direction. If you want both, you know, you kind of move up and to the right. But again, just calling out that guide and that link there because it contains sort of our— the sum of some of the experience that we've gained over the past year working with people on when to fine-tune, when not to fine-tune, and when to use some of these other techniques. So fine-tuning actually tends to be, you know, one of the later techniques that we look for when we're working with someone to try and help them build for their, their application. It's not kind of the first tool that you reach for in your toolbox. It's, it's for the people who use it. It works very well, but a lot of people are able to, due to the strength of our base models, get away with kind of avoiding fine-tuning.

And can I ask you a question about the upper right-hand quadrant? Yeah, so, um, like, you know, it's bouncing back between fine-tuning a model and then adding RAG to the training examples.

Wouldn't it—

like, what do you— what does this mean exactly? Like, and you find that people don't add RAG to their training examples? They're not adding RAG to their training examples, doesn't that mess up the production like, what, you know, doesn't that create a drift in production? Or just curious, like, what this graphic means right here.

Yeah, I don't know if this graphic is supposed to be, uh, 100% true or just kind of representing sort of the general trend here, but, um, the, uh, you know, as it sort of depends what you need, right? If you want to add more, um, knowledge or context, you, you use RAG. If you want to change how the model's responding, you use fine-tuning. And if you want to do both, then you're going to use a combination of both. I don't know if that really answered your question, but—

Yeah, I mean, you kind of did. You said, like, don't take it too literally.

Don't take it too literally. And this is kind of an example path that someone would take. But obviously, your application, you might stop here. You might not have any need for RAG at all and just sort of go straight to the fine-tuning product. Again, we really advise that you find a set of, just in general for optimizing LLMs on OpenAI or elsewhere, that you build for yourself a set of evals that, you know, make sense for you and your application, and then just work towards whatever's the simplest kind of thing to get you the performance you need on those evals. And if that's just prompt engineering, great. If it involves fine-tuning, you know, we have that offering too. If it involves retrieval or access to additional tools like code interpreter and so forth, you know, we want to give you those options. But the key thing is that, you know, what works for one person, one use case, one customer is not going to work for someone else.

Do you find emails— I think that's the most important topic in many cases. Do you recommend things to your customers who are like fine-tuning in terms of evals, like, like tools or patterns or starting points or anything, like any kind of—

yeah, I, uh, yeah, I don't have particular tool recommendations right now, but, um, the, the thing we always recommend is building your own set of evals rather than saying like, oh look, I found MMLU or whatever online, um, and I think this performs a little bit better on that. Like, that's not going to matter for you unless you're actually answering textbook questions like MMLU does. What's going to matter is, is real prompts from your sort of application and what, what the desired responses should be. I wanted to share a cautionary tale when fine-tuning doesn't work. It's kind of a funny example we saw last year. Someone took— their goal was basically to fine-tune a Slack bot that would answer questions, onboarding questions from new employees at their company, and then, you know, automatically respond to them versus having to wait for someone to respond for you. So, you know, that, that kind of falls into our what not to do category of adding new knowledge. And what happened when they fine-tuned the model against their entire Slack corpus was that the model learned the format of the responses, but not necessarily the information. So for instance, someone asked, you know, this Slack bot, write a 500-word blog post on prompt engineering. And based on the Slack corpus of data, the format that it learned was, you know, I'll do that tomorrow. And you know, you can prompt it again and say, write it now, and it'll say things like okay, which was, you know, occurred far more often in the training data. So here's where I guess that kind of previous slide on what to do and what not to do comes into play. What the model has learned here is how to respond to these inputs with the most common outputs it saw, which in this case, this you know, was sort of deferring work, and it wasn't learning the new knowledge. So the better use case, the better kind of approach to this would likely have been something like RAG on the Slack data. If you've used the fine-tuning API, this will be familiar to you. You format your data the same way as the Chat Completions API. So you have a system message, a user message, but then the only difference is here in the fine-tuning data, you're including, you know, the desired response from the assistant. So we recommend using between 50 and 100 examples. All the advice I said before applies previously, like you're trying to— one model should kind of learn one set of tasks and not a diverse set of tasks. So lots of examples of the same set of tasks. And it's also important to keep the shape of the system and user prompts that you use in your fine-tuning data similar to what you'll actually use when you call the model. So remember that the fine-tuning makes the model learn how to respond to a particular sort of pattern or format of inputs. And so if you want them— if you want it to do that, you need to make sure that the inputs are kind of similar from training off to when you're actually calling it.

I have an advanced question for you about this topic. So Hamel and I actually face this all the time. So if you have multi-turn conversations and you have like RAG and function calling and all this stuff, And like a lot of times, let's say you're using a library like LangChain, a lot of like internal thoughts, quote, internal thoughts. So like the first turn of the conversation might include internal thoughts. The second turn of the conversation won't repeat all of those internal thoughts. It'll just give like the result of whatever, you know, those thoughts were or whatever. And then like the question then quickly becomes like, well, how do you, what's the best way to prepare your fine-tuning data? Like, because, yeah, like, you know, should you include those in, like, because sometimes in production you will have those internal thoughts. Sometimes you will not have those internal thoughts. Should you just like shove both in there? You know, and then like the question becomes, oh, that feels really duplicative because, you know, multi-channel conversations can be very long and then like whatever internal thoughts and whatever. So I just want to throw like, see if you have any reaction to that.

Yeah, yeah. I mean, I don't know off the top of my head exactly what the best thing to do there would be, but we do— you can experiment with the weight parameter, which also can be added to this training file input. So it allows you to say, you know, do you want to only train, for instance, on the final assistant response and learn that format? Or do you also want to look at kind of learn the the chain of thought or the process it got to get there. In general, though, the kind of from my— oops, what not to do at some point. Well, it's on a different slide, but if you're trying to fine-tune something that's too complicated, it's sort of less likely to work. So you know, the best advice I guess is to, to make some eBells, try experimenting with the weight parameter, and then if it just isn't working for these kind of multi-turn conversations, it, it could be that it's too complicated for, um, our self-serve fine-tuning.

So when the weight parameter is 0, is that equivalent to setting the label of those tokens to like -100 or whatever? You know, like, does it— like, in the Is it basically— what is, what is happening exactly when you—

it's not going to learn any information from those messages. Um, so by default, the, the user, for instance, uh, the, the, the like user and system messages, um, uh, the weight is set to 0. Like it's learning, that's the context for what it's learning there for the messages where the weight is set to 1.

Okay, but will it still give that context to the model during training? It'll just ignore the— it just won't—

it won't be learning about how to respond or how to output that message, but it will be used like using it to learn sort of as input.

Okay, I see.

Yeah, I do believe that what we did was actually to experiment with the weight parameter, and we finally managed to get it to work, but The one question I had in mind was that in more complex examples like this, what happened was that there was no weight parameter and it was added silently. So it took us like a month of scratching our heads until we realized there's a change in the documentation and there's a new weight parameter. But my question from you would be, generally speaking, how does OpenAI decide what direction to take on these? Where's the feedback coming from? Is this your internal? Products that you're building, or is it with collaboration with your clients? And could that process be more open or more documented in the future?

Yeah, I mean, we, we get feedback from a lot of places. There's the OpenAI Developer Forum. We work with, with Customers 101, and we have account managers for some of our larger customers where we'll, we'll pull in feedback. And then honestly, events like, like this and, and conferences and talks and so forth, and just going out and talking to people, we hear requests and ideas on things we want to add and kind of, you know, collate that and, and try and decide what's important. So the weight parameter, which was introduced recently, is something that came from discussions like this. I'm, you know, I guess we could have made, we should have made more of a splashy rollout of that given the the amount it's been, or how well it's been received. So, but yeah, it's, you could expect going forward that we're going to continue to add more models, more methods, more customization to the offering. But I can't give you any specifics today. Here's a slide you maybe want to just kind of screenshot. These are a collection of some of our learnings from, from working with people over the past year on best practices for getting fine-tuning to work, some of which I've already covered, like making sure that you have fewer high-quality training examples that all are on a particular you know, a single task. I'll talk in the next slide about the different hyperparameters we offer currently. Again, the evals there on the bottom, important to make sure you're using your own evals and running a baseline and optimizing against that. And then you know, if it's important for your application to reduce latency and cost, then you can try fine-tuning 3.5 on, um, and comparing that to GPT-4 for your use case. And sometimes you'll see that the fine-tuned 3.5 is going to be as good as the base 4, but obviously cheaper and faster. Hyperparameters we offer today. Again, this slide was kind of designed to be just a screenshot. You can also look at the documentation. The hyperparameter which affects training the most is epochs. This is basically how many times is your training data iterated over during the training process. If it's 1, then the model the fine-tuning process will only see each example one time. This kind of, by default, if you don't set it, we will choose something based on the size of the dataset. So if you have a really, really large dataset, we don't want to iterate over it multiple times. But you also have control over this and can try sweeping over this parameter as necessary. If you set it too high, you're going to overfit. Set it too low, you're not going to learn enough from your training set depending on the size. Batch size and learning rate multiplier are, have a smaller impact on the fine-tuning, but are still important. And you can see some recommendations there we have on, on setting those. But again, the default will try and do the, the best thing for, for the most number of people. and if you need to change it, you can.

Are you going to talk about how to interpret the, you know, the training loss graph, whatever telemetry that y'all provide?

Yeah, I'm not an expert on that, actually, that portion of it, so I don't want to say it's wrong, but in general, you want to see that graph go down and to the right to indicate that things are working. And if it's sort of spiky and all over the place, that means maybe something has gone wrong.

Yeah, I mean, what I always— it's quite interesting, like whenever I fine-tune OpenAI model, I see it go down very quickly, like with the default every single time, and it just kind of like bounces around near zero.

Yep.

Even with just like one epoch. And it's very interesting. I'm like, oh, okay, like, should I interpret this the same as like, the, like, as when I train open models, like when I'm fine-tuning open models, is this like something special here that I don't know about, like, that I'm like interpreting it wrong. I like all these thoughts go through my mind.

Yeah, I mean, we're basically exposing the loss that, you know, we, our infrastructure uses for doing our own training. So I'd expect that to act behave as, as you would for any other loss curve. If it's going down really quickly immediately, then, you know, you might try with, with fewer epochs or a smaller training file and see if you get basically the same performance.

Okay, so it should look the same of what you expect from like OpenModels or whatever. There's nothing like idiosyncratic about it per se.

Just some examples of success stories we've seen in the past. You know, as you fine-tune larger models, the, for instance, annotation can get better, and these are both better than the base model. This, we worked with the government of Iceland. There's a case study, I think, online to do grammar correction for them. And you can see that, you know, if you remember kind of that 2x2 grid I was showing about optimizing responses, that the kind of farther right up and right you go, the better, in this case, examples or results you're getting. So with zero-shot prompting, you know, their evals were lower than with few-shot prompting, which were lower than with 3.5 fine-tuning, and then which were in turn lower than GPT-4 fine-tuning. So for, you know, if it's working well on your evals, you can kind of continue moving up that mountain and get better and better results, or moving to the right in that graph and getting better and better results.

The thing on the left-hand side of this slide is confusing. Me a little bit. Like, this is— is this saying you moved from GPT-3.5 to GPT-4 and saw an increase in performance? But it's— they're both fine-tuned, so I guess like it doesn't really—

yeah, that's what the, the graphic here doesn't have, the un-fine-tuned version, but those, those are also smaller numbers. But okay, um, I you can fine-tune larger models as well and get better performance in some cases. But on the right here, you can see that like this, just the 3.5 fine-tune is doing better than the GPT-4 few-shot prompt or zero-shot prompt, which, you know, this, this, I don't know if you can see my cursor, but this third, this middle bar graph here not only is getting better results than the ones on the left, but it's also going to be faster and cheaper. Than the GPT-4 prompts without fine-tunes.

The thing that's curious in that graph, like, why is RAG, like, fine-tuning plus RAG going down?

I don't know. Should have looked that up before including it in my presentation, maybe. But if the— it could be kind of a cautionary tale of like, if the use case here did not require you to introduce new context, then kind of adding on techniques that are not needed will actually make things more complicated and worse.

One question I have is, do you know which exact model of GPT-4 are we discussing here? Because to my knowledge, the very early GPT-4 0.6.13 is the one that is available for fine-tuning. Is that the model that has been fine-tuned in these graphs, or is it the newer Turbo or GPT-4 models also included?

To the best of my knowledge, this is the original GPT-4 because we did this engagement last year, and that's, you know, still where we have a beta program with some select partners for that. But yeah.

And I think we had the question that is relevant now. Are the newer models like GPT-4.0 going to become available for fine-tuning?

We're working on it. We're working with a select group of partners on beta testing that. And again, I can't really share timelines or exact plans, but But you shouldn't be surprised if over the next year we have more models and more methods, more hyperparameters, etc., available in the fine-tuning product.

I have a question that you may not be able to answer, so, uh, but we'll try. Um, and it reminded me because I saw GP— okay, so like one of the things that is really common is people take GPT-4 and like use the data to like fine-tune GPT-3.5 or something like that.

Yep.

Now, is it— there's some confusion. There's like tons of confusion out there whether it's against the OpenAI terms of service to use GPT-4 data that is like some data to train your own model, like an open model. Like you're not trying to compete with OpenAI, you're not trying to sell the model or do anything, but like you're just trying to train your own, like for certain use cases. Do you know if that's against the terms of service or not?

I'm not the right person to ask. I don't want to say something and get someone in trouble. I know for sure that fine-tuning or distilling a model onto an open-source model and then distributing that, you know, is against our terms of service. I don't know the answer to your particular question though about if you use it just in your own application, whether or not that's against terms of service. I can tell you that distilling GPT-4 into GPT-3.5 fully through our platform for fine-tuning is completely fine, and we have a lot of people using that. So, but yeah, I don't want to answer.

No problem. I know, I knew it was a little bit tricky.

Alex has asked a question on topic, that is, what are the advantages of fine-tuning OpenAI models over an open-source model? And when do you think it makes sense to use OpenAI against an OpenAI model?

So the OpenAI models, you know, when we move forward with the GPT-4 fine-tuning and some of the newer models, obviously those are state-of-the-art right now, so you have that advantage. We also feel that, like, the OpenAI models offer tool calling and function calling and a lot of sort of other features, but And there's some advantages to just not having to worry about deploying your own models and so forth. But, you know, there's a lot of places within OpenAI's product offerings, API, where like we're just not covering stuff that you can do better elsewhere. And so I think, you know, provided you're not violating the terms of service, like I said before, the there's options for people.

Yeah, I'm also gonna have a stab at answering that because when you try to do more complex stuff or in large scales, none of the— we have never managed to get something working as good as OpenAI, even the GPT-3.5 models. So that's my stab at it. If you really want to get to a product that works, I think as of right now, it's still— especially if you're using something like tool calling or you want to call them in large numbers and you don't want to hit API rate limits or anything like that, definitely working with OpenAI is significantly easier option.

Yeah, awesome. That's great to hear. We, we put a lot of work into trying to make everything turnkey and just work. So, glad it's working for a lot of people. Speaking of function calling, actually, this is another use case where you'll find some, some folks have found advantages from doing fine-tuning, and that is just kind of instructing or teaching the model to respond exactly with the right function that they need or the right output format. But I guess one caution with fine-tuning for function calling is that Unlike sort of, unlike when the, the context for functions and putting all the definitions, all the parameters, the descriptions of everything is kind of large enough or complicated enough in many use cases that you cannot get away from, you cannot get away from including those when you actually do your prompt, if that makes sense. So, you know, sometimes we're, we're seeing cost savings from other use cases where you have a big, a long prompt and you can fine-tune that prompt away at the chat completion time. But because function calling, you have the really long list of complex functions, it doesn't reliably work right now to kind of fully fine-tune that away. So sometimes it works, like I said. But just a caution there, we've seen this not always works.

That's interesting. Why do you think that is? Like, if you give a whole bunch of training data that has like lists of function tools and then like how, like examples of calling all of those functions and whatever, why wouldn't, why doesn't that work, do you think? That's very interesting.

I think it falls into the same category as the trying to teach a model to do a bunch of different unrelated tasks. Or I guess these are sort of somewhat related, but trying to teach, trying to fine-tune too many tasks into one model does not work as well as, or work as reliably as the same task over and over again.

So what's the order of magnitude that you have in mind when you say like too many functions, like a list of what, just curious.

Yeah, I wish I could like give you a number and say, you know, at 5 functions it breaks, but It's, we've seen it, you know, we've run a lot of evals internally and seen it fail at small numbers and succeed at larger numbers. So it also depends on kind of the complexity of each individual function and also, you know, whether you're trying to use parallel function calling, some of these other advanced features, so. I hate to be a broken record a lot of the time when I'm doing these talks and just say, you know, well, run evals and try it. But unfortunately, that's our best answer in a lot of cases.

Is there an evaluation framework that is focused purely on function calling and the intelligence of function calling that we can look at?

I don't know. There's some, there was an, the name is—

There's a Berkeley Gorilla leaderboard and that, yeah, that's really focused on function calling.

So that's, you know, some, a good set of evals for generic function calling. But again, you'll wanna try and like the best evals for your function calling are gonna be your functions. So yeah. This was actually the answer to Simon's question from earlier. But here's— this is our cookbook for the Q&A examples where you're training, fine-tuning a model to respond more often with I don't know and kind of reducing hallucinations by adding both sort of questions that you're allowed for. I think the cookbook in particular uses questions about the Olympics. So it says, you know, if the question is about the Olympics and you know about these things, you can answer it. Otherwise, you should respond with, I don't know, if it's outside that kind of knowledge base. And fine-tuning does really well at sort of reducing the number of false positives. That's all I had in terms of slides. I guess it's kind of up to you guys whether we should do some more questions, or I can run through a demo of our UI. I've got a demo of just running a fine-tuning process mechanically, how it works.

I think I know that Hamel Husain has some good questions, so we should let him ask some questions.

Yeah, I'm pretty much sure we can. There's a lot of YouTube videos on how to use OpenAI, but we don't find you elsewhere, so maybe we can use this time to extract as much as we can here. On the topic of evals, I also see a lot of people asking questions that there is very abstract that, hey, you should run your evals. Let's maybe get a little bit more elaborate into that. One question, and generally speaking, thought I have is that when speaking about evals, what I'm finding out is that a lot of people think that evals also should all be AI-based. It should all be another language model evaluating the output of— is that what you mean by eval? Or, but in our case, for example, we did the opposite. We did software doing evaluations for the language model outputs. But can you elaborate a little bit in terms of what do we mean by eval exactly?

Yeah, we mean really whatever works for you. So It's obviously cheaper and easier if you don't have to use a model to do the grading of whether an eval is correct. So maybe it's like if you're fine-tuning a model to respond in a particular format that you just write some code that checks yes or no, did it respond in that format, and can quickly tell you how well the model has done. Um, for sort of more complex questions about style, tone, whether or not the model included some details about this or that, you know, that's when you'd also need to call a grader. Um, we do this a lot internally as well. We have all manners of graders, from like simple set.contains Python function all the way to calling our latest models on the response and asking it a question, a yes/no question about whether it's met some criteria.

Okay, perfect. And I see some people asking about the IP and licensing terms for the data they provide for fine-tuning data. I know that you're not a lawyer and you may not be necessarily the best person to ask this, but what happens to the data that we provide as examples for fine-tuning data?

Yeah, so one thing to say right off the bat is we do not ever train on customer data. So any of our foundation models, there's, that's covered, you know, it's, I think it's one of the first lines of our privacy doc in our terms of service, but we do not use that data at all for internal purposes. And also you have control over the lifecycle of that data. So You can upload the data for fine-tuning. Once the fine-tuning completes, you delete it. It gets deleted from our servers. Or you can leave it on, leave it with us and fine-tune future models. It's up to you. You have full control over that. I'll, you know, point people towards our platform enterprise privacy documents, which is available on platformopenai.com, and just say that, yeah, you know, we, we have a number of customers who are trusting us with their, um, a number of customers are trusting us with sort of their application, their, their business use case, their, um, you know, their reason for existing. And we take that, uh, responsibility very seriously.

Um, one question I have is like, what, like, in order of magnitude, like, what percent of prediction requests get routed to fine-tuned models? Just out of curiosity, like, how popular is that segment of models? It'd be really interesting to know.

Yeah, I don't have the numbers in front of me, but I would say pretty confidently it's like less than 1% of our API users are using fine-tuning. You know, when I said at the beginning of this, the presentation, that like fine-tuning is one of the last tools you should reach for, it's, you know, serious about that. It's kind of, it's what you use when you're at the final stage of your application and you're trying to optimize cost or latency or it's what you use when the few-shot, many-shot prompting and your standard toolbox of tools isn't working for you. And to be honest, like, you know, few— I guess those tools work pretty well. And so, uh, you know, not everyone needs fine-tuning. Um, but for the people who I do end up need fine-tuning, you know, they're, they're really happy with it, and they wouldn't have been able to, to get there without it. So, um, but, uh, you know, it's, it's a fraction of, of, of what we do. It's kind of the long tail of, um, you know, tends to be larger customers, um, with a lot— with more, you know, LLM expertise that are working on fine-tuning, less so the, you know, people who are using chat completions and getting enough out of that on its own.

Are you finding that a lot of people who were fine-tuning, as you continue to release more and more powerful models, they pivot away from it completely and they say, okay, let's stop fine-tuning. Let's improve our prompt engineering, RAG, whatever. They were able to get off? Or just out of curiosity.

Yeah. Sometimes the diff between a 3.5 fine-tune and the 4 base model is large enough that it's worth sort of abandoning your fine-tune and just going to one of our new, you know, 4.0 models. Sometimes when that diff is small though, people will stay with the 3.5 fine-tunes because it's cheaper and faster. So yeah, it really depends.

That answer worries me a little bit, maybe because we've been burned by Google so many times, but from a sustainability perspective, as we continue to invest in our infrastructure for fine-tuning, can we count on fine-tuning being available and being pushed forward, but by OpenAI as well?

Yeah, um, uh, we, you know, continue to support any application, or like applications that are working well, um, and, uh, I don't want to leave anyone high and dry, um, with work invested. We recognize that like fine-tuning is different than than Chat Completions, and that it's not something, you know, there's the work you've put into preparing your dataset, there's the work you've put in and the money you've already spent with us actually training the model. So there's, you know, a higher switching cost there, and we recognize that and respect that.

That's great. There's a lot of speculation about agents and what language models can do to enable agents finally becoming reality. Are there some flagship products that you guys have seen on OpenAI that you're like, this is a great use of OpenAI that ended up creating a good agent? Or generally speaking, some of products that are doing more than just simple completions, maybe they're great use cases for function calling or great RAG use cases that you guys, you maybe can point us to take a look at.

Um, I'm the wrong person to ask for agent stuff. Uh, we, we have another team that works on the assistance API and, and a lot of the, the tool calling. I, I, I guess speaking for myself and just kind of personal interest, I find the, like, the, the Devin, the GitHub Workspaces, uh, GitHub AI Workspaces stuff really cool. Um, basically like the ability to, uh, create a plan, um, uh, engage with a bunch of tools and work, have, you know, different LLM threads working together to solve a more complicated coding task. Um, I think that's gonna be really cool. Um, but, uh, uh, yeah, more, more broadly than my kind of personal interests, I, I, I'm the wrong person to ask that question.

Is there, is there a tension between like Okay, like if OpenAI starts to offer higher-level services like agents or functions that you can call that they will execute for you in fine-tuning, because then where will you get the data for fine-tuning if it's not— you don't log it yourself or whatever, you know what I'm trying to ask? Like if it's like a— if it's a service, like for example, already like the assistance API, You know, like, y'all will sort of keep track of the conversation for you. You know, and so like, you have to be careful to like capture all the data yourself. You're trying to reuse it for fine-tuning. And then it's also somewhat not clear sometimes. Like, you have to, you have to like wonder, okay, like, what is the Assistant API actually doing? Like, how is it being cut off? Or is there, is it being summarized or doing something fancy? Like, you know, Um, and so like, is there a tension between like fine-tuning and then like these products that are abstracting some of what's happening away?

Um, I don't think so. I mean, you can use fine-tuned models within the Assistants API, for instance, um, and for the same reasons that you'd use them in the Chat Completions API, uh, to craft your outputs in a certain way. Um, the Assistants API is really helping with, uh, well, one, giving you access to more tools, but two, also just managing the, the, the context for you. So rather than needing to store, you know, previous turns in the conversation on your own systems, uh, it's quicker to get to an MVP or a product, um, through, um, letting us manage that.

Yeah, what I was asking is mainly like, okay, if the Assistance API is truncating context, then, you know, when you when you fine-tune your model, I suppose you should truncate that context too. Like, you have to, like, know what it's doing for you.

Yeah. Another question I'm not qualified to answer, but I don't know the details on the context truncation within the Assistants API right now, so it'd be hard for me to answer that.

Let me ask another question regarding function calling. Playing with function calling, our understanding is that we provide functions as JSON schemas, but down the road, before they are being processed, they're actually translated into a TypeScript-like format before the language model actually processes them. Can you explain that a little bit? Is that true? And is that something that enables us to do something like maybe providing easier function definitions for the language model?

Yeah, they're all great questions. A lot of them I'm not qualified to answer. I can't tell you, I can't tell you the details, because I don't know them. But I can tell you that, like, you know, our models only understand tokens, which are just integers. And so the JSON schema, whatever function, every everything that goes into the chat completions, eventually turns into tokens in sort of our own format. So, um, uh, they—

like I said, I can't show the details because I don't know them, but also I think there's all these funny experiments like where you can try to get the system prompt or the prompt, and then like people fill around and say, oh, okay, like the function, the function, like the tools are being like flattened into these like TypeScript definitions.

I'll, I'll tell you that I think in general a goal with the API and with ChatGPT is that like the experience that we offer through the, the function calling input or through kind of prompting ChatGPT, the default experience without tricks is the one that we're trying to make the best. So like, you know, that earlier this year there was that thing where like you could you could tell ChatGPT you were going to tip it $20 and it would give you a better response. You know, we then trained it to just give the good response without having to promise to tip it. So, you know, our kind of, you know, I appreciate that people are like interested in trying to get that extra little percentage of performance by understanding what happens under the hood. But like our goal is that the default, what you provide us is just the one that works the best without needing to do any kind of weird, weird runarounds and tricks and things.