Fine Tuning LLMs for Function Calling w/Pawel Garbacki

Yeah, so thanks for having me. I'm Paweł from Fireworks AI, and, you know, in this presentation, I will give you some tips and tricks around fine-tuning your models for function calling. Next slide. So let's start with what is actually function or tool calling. So the way I like to think about it is that, you know, in some cases, you need to give your model the ability to interface with the external world. So, you know, one example use case is where you want your model to get access to the information that wasn't available in the model training dataset. So, like, you know, that includes in particular the types of information that is available in real time, as in this specific example where we want the model to help to retrieve the current stock price of NVIDIA over the last year. You know, this type of information requires access to you know, basically the stock price time series over the recent time. And since that type of information, you know, it wasn't included in the pre-training dataset, and we need to give the model the ability to pull this information from elsewhere. You know, another like very common use case for function calling is orchestration in multi-agent systems where you have, you know, number of tools that the model may act in order to assist the user, and the function calling functionality provides you with a way to do that. Next slide. So let me guide you with, you know, some of the decisions you need to make when you decide to tune your model for function calling. And one of the more important decisions is the objective, right? And the objective you pick is going to affect a lot of things like, you know, the data you need to prepare, the amount of training data, the overall cost, complexity of the fine-tuning and the complexity of use. So in general, it is recommended to pick just the right objective that is, you know, required for your use case and try to keep the objective, you know, as simple as possible. Previous slide, yeah. So typically, like, you know, the most common and also like the easiest objective is what I call single-turn forced call. Some other people also call it the routing use case. So in this case, we have access to a number of functions, and the user will provide us with a single instruction, and that instruction will be mapped to a call of one of those functions. And basically, the objective of the model is to pick the right function and to pick the parameters for that.

Why is it called a forced call?

Is it because you're—

yeah, sorry.

Yeah, it's called a forced call because we are forcing a function call, right? So the model won't be able to answer a question like, how are you today, right? Because that would be just like a natural language response. But in this case, we are kind of forcing the function call. So the response is also always going to be a function call.

I see. Thanks.

Next slide. OK, so a slightly more complex objective is parallel function calling. So in this case, we are also forcing function calls, but instead instead of calling one function, we may call multiple functions, but those functions are kind of independent of each other, so they can be called in parallel. And so one example here is like, let's say we want to get the stock price of two companies, NVIDIA and Apple, and we have access to a function that can take only, you know, one ticker at a time, so we need to split this invocation into two function calls, but they are independent of each other, so they can be called in parallel. Next slide. Okay, so let's complicate it. So you can also have cases where you have dependency between functions, right? So like you may call one function and that will return some response and that response will need to feed into another function call. So I call it nested calls. So as in this example, after getting the stock price of NVIDIA and Apple, we also want to plot those prices. So we basically have two functions, one to get the prices and the other one to, you know, interpret those prices and plot them. So as you can see, like in this case, we are entering this like a multi-turn, you know, scenarios where we need to call the functions sequentially starting with the nested function, and then we call the outer function to interpret the results of the nested calls. Next slide.

And let me ask a quick question on this one. So if I use this as training data, I don't see where the model is learning that when it plot— when it calls the tool for plotting— oh, I see. The tool is returning this 120 and 121. I see. And then that's what's getting plugged into the plot call. Okay. Sorry. I was just missing something.

It's a good question. It's a good question. So here, maybe I should have specified it earlier. I'm using the terminology, which is following the OpenAI APIs. And like typically in this case, you have kind of like 3 roles. Like you have the user role, you know, which is usually the client interfacing with the model. The assistant is the model itself, so it is generating the function calls. And then you have like a special role called tool, which is basically like used also on the client side to feed the result of the function call back into the model, right? So in this case, you know, we have the user issuing an instruction and then we have the model generating function calls. And typically those function calls are interpreted on the client side by basically invoking them. And feeding the results back into the model, and the model may decide to follow up on that, as in this case, calling another function, feeding the results from the initial function calls into this second function call.

Yep.

Okay, and finally, probably the most complicated, or one of the more complicated use cases is the one where we want to print conversations with the function calls, right? So we may think about it as, you know, the user is having conversation with the model, and, you know, in certain situations, and the response from the model may involve a function call, right? So I call it multi-turn chat use case with optional function calling. So as in this case, like, you know, the user may inquire about, like, what's in the news today, and with which assistant will respond with a function call to pull in the training use, and response from that will be fed back into the assistant who is going to summarize the response, to which user may, you know, follow up with, you know, additional inquiries related to the previous responses. So this is typically like the most complicated use case to tune for because it basically involves multiple objectives. So like the model needs to be good at both you know, responding in natural language and having like a natural language conversation with the user, which is interleaved with function calls. Next slide. Okay, so let's switch gears and call and talk about the function call token. So what is the function call token? So in general, like, you know, the way that the client interfaces with the model is that, you know, the model is going to return a response and the client will have to interpret it. And, you know, especially in the cases where we have complex objectives like the Multiterm General Chat with Optional Function Calling, the client will have to tell the intent of the response, and the intent could be like a function call, or maybe like no function call, or maybe like multiple function calls. So it is generally advisable to make the output of the model structured in such a way that it is easy for the client to parse it and to tell, you know, when we perform a function call and like, you know, if the response includes like a mix of natural language and function calls to also tell where is the boundary of the function call. So the general advice here is to introduce like a special token, like a single token that is going to prefix the function call. So the reason to do that is actually threefold. So the first one I already mentioned, like it makes parsing of the response easier. But there are also like more— so, you know, typically the modern inference services give you the ability to stream the response back. Instead of like waiting for the response to come back fully and only then revealing it to the client, you may want to stream the response chunk by chunk. However, like when it comes to function calling, you probably don't want to like stream the response token by token, but instead you want to, you know, when you enter the function call mode, you want to wait for the entire function call signature to be generated before you return it back to the user. Otherwise, like, you know, it's pretty hard to parse like, you know, partial function calls. So having like a single token telling you that, okay, you know, now we are switching into the function call mode. Makes it easier to implement streaming generation. Also, there will be like another concept which I call constraint generation that I will be discussing later in this presentation. But just like I said previously, it's a way to enforce like a certain schema of the function calls. And also by having this way of, you know, switching the model intent to to function call, it makes it easier to basically enable the constraint generation mode. I will come back to this concept in a few slides. Next slide, please. All right, another decision you need to make is to pick the syntax for defining function calling. So I would say that, you know, based on what I have seen, there are two major contenders here. One is to perform a function call through the Python syntax, so basically generate like a Python function call signature. And the other one is to generate a JSON schema, so basically have like, you know, JSON structure that describes the function name and the parameters. And so, you know, so far in this presentation I have been following the JSON schema syntax. But, you know, the Python, Python function signature generation is also a pretty good option. So when considering the trade-offs between those two, here is what you should take into account. So generally, it is a little bit easier for the model to generate Python function signatures. So the main reason is that, you know, modern LLMs have been trained extensively with Python and to understand Python. So by relying on generating generating the function calls in the Python syntax, we're kind of piggybacking on the model's like internal understanding of Python. However, like the problem with this approach is that, you know, it is not very common for like, you know, the existing code in Python to perform, you know, calls of functions with very complex parameters structures that are inline in the call. So in that parameter, right, typically what you do, like in Python, you would create like another class that, you know, defines the type of this parameter, and then like you would basically set the values of that object like outside of the function call and pass it to the function call. However, like, you know, here we are generating basically like a single line invocation of the function, which makes it like pretty unnatural to pass in deeply nested parameter types. So if your use case involves deeply nested complex parameter types, you know, signature may not be the right choice for you. JSON schema, on the other hand, It's better for those, like, you know, complex deeply nested parameter types. It is also a little bit easier to generate with this constraint generation concept, which I will discuss in a slide or two, where we are basically, like, you know, specifying, like, a grammar of our function and making sure that the model follows this grammar. And it also has the advantage that if you want your model to be exposed through an OpenAI-compatible API, which is pretty common nowadays, it may be a better choice because, you know, OpenAI models like GPT-4 are following the JSON schema syntax. So if you want to, like, you know, stay compatible with the leading offerings, you may want to pick the JSON schema option. Next slide. Right. Okay. Another consideration you may take into account is around preserving the capabilities of the, of the model you are tuning on top of. So, you know, if you compare the quality of the models that we are getting nowadays compared to the quality of models we have been getting, let's say, 6 months ago, we have seen like a very significant improvement in instruction following capabilities. So this is mainly coming from the fact that, you know, companies that tune those open-source models have been investing pretty heavily in creating very high-quality alignment datasets. And, you know, maintaining and curating those datasets involves significant effort and also costs a lot of money. So, like, for instance, Llama 3 has been aligned with 10 million human-curated samples, right? And because of that, like, Llama 3 is, like, really, really good at instruction following. So if you are tuning your function calling model to mix function calling with general chat, it will be a little bit of shame if we would kind of override the instruction following capabilities during our fine-tuning process. So what we are doing nowadays when tuning function calling models for those more complicated objectives, we try to preserve the existing instruction following capabilities of the model we are tuning on top of while adding function calling on top of them. First of all, you should pick the instruct version of the model, not the barebones version of the base model when you are doing tuning. Also, in order to preserve—

Can I ask you a question about that?

Sorry.

So are you saying, like, OK, when you're fine-tuning a model for function calling, you recommend fine-tuning on top of the instruction-tuned model instead of starting from the base? Did I understand it correctly?

Exactly. So it kind of— yes, that's what I'm saying. However, the caveat here is that it kind of depends on the objective you are tuning on. So that goes back to my previous slides. If you are tuning for a forced function call objective, right, where you know you are going to call a function, you are not going to, like, follow instructions or, like, have general conversation with the user, in this case, it probably doesn't matter, and you can pick the base version of the model. But if you are tuning for a more complicated objective that interleaves general conversation with function calling, you may want to pick the instruct variant of the model. I see.

Thank you.

Right. So, you know, in general, in order to preserve as many of the instruction following capabilities as we can, it makes sense to reduce the amount of training data, right? But then, of course, like, it may have an impact on the quality of the model. So if we don't use a lot of training data, like we would better make sure that the data that we are using is of high quality. So the general recommendation here is to basically like reduce the amount of training data, but at the cost of preparing higher quality data for our tuning. Next slide. Right, so another consideration is whether to do full weight tuning. Versus LoRA tuning. So based on my experience, and I guess the field is kind of divided here, it depends who you ask, but based on my personal experience, LoRA is good enough for tuning models for function calling, especially in this limited data regime that I mentioned in the previous slide. So since we won't be using tons of tuning data, it is actually better to use LoRA because it has fewer parameters that we need to converge. So actually, when you tune with this extremely low data volume regime, by using LoRA, you can actually paradoxically end up with a model that is of higher quality compared to the LoRA. I wanted to point out that may not be obvious to to people is that if you're going to fine-tuning your models, you will be going through a lot of iterations. If you go through a lot of iterations, it is actually better to have a setup that lets you, you know, churn through the different versions of the model faster. Tuning LoRA is way, way faster than performing full-way tuning. It takes less resources and it also is cheaper to host and experiment with. Next slide. So why is it cheaper to host LoRAs? So it's not always cheaper. It is cheaper if your inference service has, you know, certain facilities that enable efficient LoRA serving. So in particular, the inference solution that we have developed at Fireworks, it has the ability to serve multiple LoRAs. Very cheaply. So we are able to load hundreds, if not thousands, of LoRAs per user account, and we charge you users only per token. We don't charge per memory footprint. So the reason why we can do that is that, you know, instead of merging the LoRA weights based, you know, based into the base model, we are keeping them separately. And since, you know, the LoRA weights are way, way smaller compared to the base model weights, and we are able to greatly reduce the memory footprint. And also at the inference time, we're able to do intelligent batching where we can, you know, batch requests that go into multiple LoRA adapters as they pass through the base model together. So we are basically able to do, you know, larger batch size inference for the base model part of the model. And we can use smaller batches for the individual LoRA adapter weights. But inferring through the LoRA weights is cheaper than inferring through the base model weights. So the—

I have a question, if you don't mind, while we're talking about Fireworks. Now, you showed us a lot about generally what to think about when you're fine-tuning. For function calling. Is, uh, is there a paved path on Fireworks that allows that kind of— yeah, like for fine-tuning with function calling, like is it easy, like, you know, yeah, is there some product that does that?

I'm actually going to reference a family of models that we have tuned for function calling when it comes to fine-tuning your own models. And we have a fine-tuning service that you can use. Currently, we do not have a recipe specifically for function calling, but you can use a general recipe with your curated dataset to tune a model for function calling. And, you know, as I said, you know, our platform is built in such a way that we encourage people to experiment with multiple variants of the model with a very limited cost, right, which is like super, super important. Like, from my own experience, it takes like, you know, in order to fine-tune like a really high-quality function calling model, you need to go through hundreds of iterations, right? Like, and you need to modify things like the, you know, the training data, different hyperparameters. You may want to iterate over your benchmarks and so forth. So having the ability to move really quickly across the iterations and having the ability to host multiple versions of the model at the same time and kind of comparing them side by side is pivotal in being able to create a high-quality model. I guess this whole thing applies not only to function calling, but in general to fine-tuning any kind of model for reasonably complex objectives.

Makes sense.

Next slide. Yeah, OK, so going back to the constraint generation I referenced previously. So constraint generation is a mechanism that allows us to reduce the hallucinations in the model output, right? So when we are performing a function called generation, we know the signatures of the functions because they're kind of given upfront in the API. So we can use this knowledge to guide the model to basically force it to generate the output in a certain schema, right? So when the model decides to call function to get the stock prices of a company, like, we know that that function takes like a single parameter and that parameter is of type string, and then it has name ticker, right? So we can basically like, you know, guide model generation, um, with this knowledge of the schema.

And does Fireworks offer a constraint generation thing on their product?

Okay, yes, we have, we have constraint generation support, um, and you know, and how do you specify it?

Is it some kind of grammar that you provide, or like how do you formulate?

Yeah, so currently, currently we support it for, for a function calling only, so you will have to basically provide the schema of your functions, and based on the schema of the functions, we are going to extract the grammar and enforce the output to conform to it.

Out of curiosity, are you using some kind of open source constraint generation tool like Outlines or something, or did you make your own?

Something like that. So our solution is inspired by certain context-free grammar open-source solutions, but we have customized it pretty heavily. So it's not just like, you know, plugin use of an existing tool. It's a heavily customized version of basically multiple open-source tools. We took like the best out of multiple, um, available solutions.

That's really fascinating. That should be a talk on its own.

I agree. Um, yeah, so, so, you know, the grammar gives you the ability to, um, reduce the hallucination. So many times, like, even eliminate hallucinations. But also, it actually speeds up the generation. The reason why it can speed up the generation is that You know, like for instance, in this example, right, like we have one field calling temperature. So we kind of know that, you know, this function takes only one parameter that has the name temperature. So let's say there was a token generated for temp, right? So we know that, you know, the next token is going to be like autocompleting temperature, right? So instead of like, you know, running the generation for the subsequent tokens through the model, we can kind of autocomplete them from the grammar, and this way we are kind of short-circuiting the generation and making the generation faster. Next slide. All right, so this is my final slide, and this is probably like the most important tip I have for you guys is to work smart, not hard, and actually not to fine-tune for function calling unless you have to. There are a lot of great open-source models for function calling, including one that we provided at Fireworks. And in many, many cases, those models are good enough for a lot of use cases. If you are going to be tuning for function calling, be prepared to put like, you know, sweat and tears into it. Like, it's not easy. It definitely— it's a process. It's an iterative process. It's going to take time. It's going to take effort. It's like very rewarding, but like in many times, you know, when you are on a deadline, it's not required, and you can get pretty far. I have a question about this open source. Sure.

When I go to fireworks.ai/models, And I go into that webpage and I go to language models and I try to filter by this FireFunction model. I only get— we only see v1. Is like v2 some kind of like alpha thing that is not publicly available yet?

Yes. Yeah. You did your homework. Yeah. So we didn't officially release it yet. So this is actually the first time. I am mentioning it externally. We are almost ready to release it. We'll be releasing it in the next few days. We have been testing the release candidate pretty extensively with our Discord users for the last few weeks. So I'm pretty confident about the quality of the model in the wild. So the FireFunction v2 is like not discoverable through the UI, but, you know, this link that I pasted here should work. And the best part is that it is free for the time being, so feel free to play with it. You know, send us your feedback and join our Discord. We're always happy to hear from our users, and your feedback helps a lot.

And is it— what is it based on, Llama or something, or can you—

yeah. Sure, it is. Yeah, it is based on Llama 3, uh, 70B.

So, um, when you train these, there are many things that we fine-tune on. We're collecting the data while it's actually just naturally occurring. So there might be some data set of questions and answers that we can find in those, just in the natural course of things they get generated. Um, with function calling, that tends not to be the case. And so I'm interested, when you create a Fire Function or in other examples you've seen, what is the process that you use to bootstrap getting a bunch of examples to fine-tune on? So I think there's two parts. One is what, what's the process to bootstrap that process? And then after that, you will have many function calling conversations where there's good results and many that have bad results, it's calling the wrong function. And so what are you doing on an ongoing basis, or what have you seen on an ongoing basis to curate so that you have another, a better dataset for future fine-tuning?

Yeah, that's a great question. So it is like, you know, multi-stage process. So like probably a good starting point is to look into some of the open source datasets for function calling, and there are a few out there. One of the problems with those is that they are typically be like, you know, very focused on like one type of— we do, for instance, like we wanted to try to approximate the conversation capabilities of GPT-4 mixed with function calling. So there are like not that many high-quality datasets that, you know, contain those like more complicated use cases. So I don't think that you will be able to fine-tune your model purely based on the existing open-source datasets, at least not for the most complicated objective. So you would have to indeed invest time in building your own datasets. Some of the tips here is that, first of all, it's kind of important to define the types of data or the categories of data you are interested in. So it should be more like parallel function calling, nested function calling, like, you know, how many turns is it going to be many, like single turn or like, you know, I don't know, more than 10 turns. Another thing to keep in mind is, you know, how many functions do you want to support at a time? This is like another important parameter. You know, for instance, you know, in some cases, like people want to have support for up to 5 functions. And this is like a very different story between like tuning for 5 functions versus tuning for like 50 functions. And so it is like really, really important that you kind of have some, you know, objectives in mind or some sort of like, you know, boundary cases for the use cases for the model. And you need to make sure that you have a representative data in your dataset. Actually, one pretty good source of data are also existing multi-agent systems like AutoGen, for instance, right? So you may look into those and typically come with some tutorials that, you know, plug in the function calling model. And typically, you know, those multi-agent frameworks, they have, you know, pretty complicated scenarios involving like, you know, different agents with different roles and also pretty complex system prompts. But I would say that like, you know, by far the most or the hardest dataset to get is one way of complex system prompts. And like paradoxically, this is also what we have seen, you know, among the common use cases among our users, right? So, you know, you know, there's like a pretty big difference between like this clean textbook example, you know, get stock price, right, which is like requires almost no system prompt, and the like real-world use case where you may want to say like, you know, okay, I have those 5 functions and that they can be very close to each other. One can be Google search, the other one may be Yelp search, right? And then I'm looking for restaurant. And in the system prompt, you may want to provide some guidance around which of those functions should be called in which case. It is super hard, not only in the function calling domain, but in the general fine-tuning dataset domain to find high-quality open-source dataset with complex instructions. And this is where, like, a lot of effort on our side went into to generate some of the synthetic datasets with complex instructions.

I have one more question that I have, and then when I— if you have time to stick around, we've got a bunch of other questions. So when you are generating text that gets shown to people, the real-world impact from an adversarial input is not really that great. Like, if you say, if someone puts something in and says like use a curse word, then the worst thing they could see is a curse word. But if you allow someone to make arbitrary function calls or function calls from, from some set, the security implications are obviously much greater. Are there any guardrails or guardrails best practices that you've seen that you quite like?

So in general, almost like all of the use cases of function calling I have seen are around read-only functions. I actually haven't seen like a lot of use cases around, you know, function calls that modify data. So like for instance, if you are like, you know, accessing your database, you will have, you know, functions that correspond to select statements but not update or delete statements. Of course, it doesn't mean It is not a solution, and it also doesn't mean that it has no risks because it can still access, you know, some sensitive information this way. But yeah, in general, I think this is like a problem that we will need to address at some point, like where the function calling, kind of like multi-agent systems, like become a little bit more ubiquitous. But I would say that at this point, like I don't think we are at the level where it matters that much to some extent because people are still kind of focused on a little bit simpler use cases and basically not exposing model to very sensitive APIs as opposed to trying to fix it in some way. And I guess one way to do it would be to include precise instructions in the system prompts. And that kind of like goes back to the lack of good training dataset with complex system prompts. But yeah, I definitely acknowledge it as a problem. And, you know, I'm also going to admit that we haven't looked very heavily into this space yet, but I believe that, you know, as those models and multi-agent systems become more and more popular, like, there will be time where, like, you know, we will need to look into it.

Yeah, I've in the past worried about, even with read-only functions, some, like, denial of service attack. Can I get this thing to run, you know, some expensive operation very many times? Who knows? We've got a bunch of questions. So the first from Wade Gilliam: What should the prompt template look like for all these use cases where we are fine-tuning an open-source model for single or multiple tool calls?

Right. So in general, the system prompt is your choice, but there are some guidelines. So for instance, you know, in one of the slides, I was proposing to tune on top of the existing Instruct model and try to preserve as many of the capabilities of this model as possible. So if we are doing that, it's generally better to stick to the prompt format of the Instruct model, not to change it too much. So, you know, in particular, you know, for something like FireFunction v2, like, we didn't even introduce a new role for tools, but we rather, like, you know, reused the user role, like, try to kind of, like, minimize the differences between the Instruct model and the model we're tuning for. But yeah, in general, like, you know, the structure of the prompt, like, doesn't, you know, matter as much in the cases where you tune for, like, a single call, single turn forced calls, because, you know, the only thing that matters here is that, like, you know, you have the correct prefix for the role. Like, you typically have like two roles or like three roles, maybe like one system, one user, and then you have the assistant response. So in this case, the format doesn't matter too much. But yeah, when you are entering this area of like multi-tenant chats, it is important to pick the right prompt format.

Uh, I'm taking the questions from most votes to least, so I'm going to bring up one, but I think you've already answered this. So, uh, this was— someone asked very early in your presentation, do you treat the tool as a user or give it its own role, and does message format matter? And I think you showed that it's got its own role, which is different from, um, from the user's role. And then it says, does message format matter?

So I think message formats in general, like, matters in the sense that, you know, as I mentioned in one of the slides, we need an easy way to parse the function call, right? So there are some cases in this more generic use case of mixing function calls with a general chat where, like, a single assistant response may contain, like, a mix of natural language sentence or sentences and function call. For instance, you ask a model, can you give me the stock price of NVIDIA? And the model tries to be very nice. So instead of just dumping the function call on you, it will tell you, sure, I can do it for you. And here is the function call. And then we are going to inject this special token to indicate that now it's time to switch the mode to function call generation. So yeah, it is it is kind of important, you know, to make it, make the format so that it is easier, you know, for the client to parse. And also, you know, if it's easier for the client to parse, it is typically also easier for the model to generate.

Okay, we've got another question from Wade. I'm going to make a slight addition to this. So it says, what is the most complicated function calling example you have ever seen? Successfully fine-tuned? And then I'd also be interested about what is the most complicated function calling example you've seen just with prompt engineering?

Sure. So instead of the most complicated cases I have seen being successfully tuned for, are the actually cases where GPT-4 breaks. So one example here is that, unless it got changed in the last few weeks, but as of a few weeks ago, the OpenAI API has a limit on the number of characters you can provide in the function description, right? So like each function has some like schema that you have to follow in order to define a function. And one of the fields in that schema is description. So you can describe what this function does. And they have some limit on the number of characters you can put in the description of this function. So if you have a very complicated function, the description may be longer than this limit. But what we can do when we are doing fine-tuning of our models is, you know, we basically can remove this constraint. So like we can, you know, make the descriptions as long as possible. And, you know, to make it like even more concrete, we have seen cases where, you know, people have functions with enum values that are very, very long. Like for instance, like you may have, let's say, 1,000 enums, right? Unless the names are super descriptive, you may want to actually include some information about the meaning of those enums. The JSON schema that OpenAI supports, it doesn't have a dedicated field to each enum value. Typically, people would put it in the description of the function or description of the parameter. That's where the longer character limit is helpful. In case of the other questions, so like what was the most successful and like the more, the most complex case that we were able to prompt the engineer for, so I would say that, you know, that's where the instruction following capabilities of the original model come into play, right? So I have seen cases where maybe the function is not super complicated and maybe there are not that many functions because typically the more functions you have, the more the prompt engineering starts breaking and also the more Terrains you have, the prompt engineering starts breaking. But if you have, let's say, a case where there are not too many Terrains and maybe the functions are not too complex, but the instructions are very complex, right? So you may want to have a very precise algorithm saying no, in this case, you know, call this function, in that case call some other function. Yeah, so that, I guess that would be my experience is that if it's not that much about the following of the schema of the functions, like in the parameter values, if it's more about like very complicated instructions when to call which function, then like prompt engineering in some cases maybe even like more successful than going through a function calling model.

Yeah. Um, there's a question. Um, I guess you've mostly answered this between this slide and a previous comment. So someone asks, any examples you suggest for function calling datasets and evaluation? So you have a bunch of evals or benchmarks on this slide, but other suggestions for function calling datasets, potentially multi-torn, and for function calling evaluations, right?

Yeah, so there are, so there are datasets out there. So there's the GLaive dataset that is of pretty high quality, but it has also like pretty limited coverage of different use cases, but it is like a sizable dataset of pretty high quality for the fine-tuning. In terms of the evaluation datasets, so, you know, there are some datasets from Gorilla from Berkeley and, you know, some datasets from Nexus and that you can use. One caveat I would mention here though is that, you know, those datasets are also like oftentimes kind of very clean. So, you know, there is like a very clean use case where, you know, you have, let's say, functions of like certain complexity. Like, you know, for instance, the Gorilla dataset, it contains rather simple functions without too many like nested parameters. And it also, like, you know, requires the model to follow the Python syntax. So I think that, you know, in general, what we do is we start with those publicly available datasets. But if you— okay, let me put it this way. Like, if your model is, like, doing good on those datasets, that there is a high chance you can, like, use one of the open-source models. I mean, things get pretty hairy where you have some very special use cases where the open source models cannot be of help. And typically in those cases, you would need to invest in your own evaluation dataset in order to cover those use cases. But yeah, there's definitely, there are quite a few datasets that you can use to get yourself started.

Uh, I'm gonna go back to questions in a second. As we look at this slide, I haven't said congratulations yet. This is— these are like really great metrics, beating GPT-4o across, um, many benchmarks. So this is—

yeah, this is going to be number— I mean, this is right now number 1 on the leaderboard, on the Gorilla leaderboard. I mean, not that you should get overly excited about any one leaderboard, but I mean, it is—

it's pretty cool.

Yeah, yeah, thanks, thanks. But yeah, you know, as I, as I said, like, you know, we kind of wanted to, like, instead of, instead of like building our own, uh, you know, benchmarks, like, we kind of wanted to improve the, uh, let's say, legitimacy of this model by like using someone else's datasets. Um, you know, there are also like, you know, a few things to to pay attention to. So for instance, if you look at the Gorilla leaderboard, like there, like, you know, for even for the same model, like for instance, for GPT-4, there are kind of like many variants of the benchmark. Like there is one that is using the OpenAI function calling API, right? So going through the JSON schema. And then there is like another version where like they just like do some prompt engineering of the GPT. And actually, interestingly enough, like some of the GPT-4 versions with prompt engineering do better than going through the API. And that like kind of goes back to one of the slides I had there where I was pointing out that generating Python syntax is easier to the model than generating JSON schema. But then like you're losing some capabilities like the grammar enforcement and so forth. So I think it's good to look at the benchmarks to have some rough estimate of the capabilities of your model. But I think it's really, really important to just try things out. So when you do the model selection, just don't go to the leaderboards and pick the top model, but rather, I don't know, pick the top 5 models and try them out on your use case and see which one does best.

Nice.

We got another question here. What's a good base model to fine-tune off of for function calling?

So I would say that Llama 3 is definitely a good model. The FireFunction v1 was actually tuned on Mixtral, and FireFunction v2 was trained on Llama. And we have seen like a pretty big, you know, of course, like we did a lot of changes including the dataset changes, like, you know, some of the hyperparameters and so forth. But like even just switching the base model, we have seen pretty big improvements. Also, like, you know, I would say that the model selection also depends a little bit on the objective. So like, you know, for instance, if you are, if you are picking one of those forced function call objectives, and if you do like single-terrain, like you may actually, and you are okay with a Python signature generation, you may want to pick one of those, you know, coding models. Like, you know, Llama has like, at least Llama 2 had some, you know, model that was tuned specifically for Python code generation. So in this case, this model may do better. Also, like, you know, we just start, we are starting to experiment with it because this is like a pretty new model, but the QN, model seems to be also showing a lot of promise. So I don't have like a conclusion here yet, but based on some like early experiments, it seems to be doing pretty well for, for this use case.

Yeah, there's a question a little lower down, but it's one that, um, my impression is that this is actually a quite hard problem. So, uh, If you have a long chain of calls, what is a preferred way to retain memory of state across that chain of calls?

Okay, so this is like, you know, a very like open-ended question. So, you know, in general, if you do have, you know, very long conversations, it may be also better to pick a model with longer context, right? So for instance, you know, Llama 3 has only like 8K. Context, right? So it may not be like the best use case for very, very long conversations, but then like, you know, you also need to make sure that you have enough of the long conversations in your tuning dataset to kind of make sure that the model can capture this property. So this is the, and this is kind of like the easy way out, right? But of course, you know, you can use your your imagination is the limit, right? So there are of course like things you can build, like layers you can build on top of your function calling model. So, you know, OpenAI has this like memory layer, right? But you can, you know, you can build like your own version of it where you can, let's say, as your conversation is like going out of the context window of the model, like, you may still want to use some sort of approach to, you know, for this user question, you will try to, like, find the messages from the previous conversation that, like, fell out of the context window and kind of, like, inject them back, right? So, like, basically that would fall under the umbrella of intelligent conversation pruning, right? So typically The most, like, naive approach to, um, you know, working around the context size limitation is that you take, like, the top messages, like, you know, the early messages, and then you take the tail messages and you kind of remove the messages, like, from the middle, right? Because typically, like, you know, the beginning and, like, the end of the conversation are the most relevant parts. But of course, you can have, like, a smarter algorithm to do some sort of, like, you know, semantic matching between, like, the latest user instruction and, you know, the middle messages. Like, I don't think I'm going to, like, you know, invent anything super smart on the spot, but I guess, like, there are definitely a lot of opportunities. And, you know, if people are interested in exploring this area, like, how to do intelligent message selection, And that will be like potentially a very high impact area.

For what it's worth, there's something that I have on my backlog to explore, which is, um, instead of keeping track of the whole conversational back and forth, uh, at the end of the— at the end of each turn, basically have the model summarize like, here is the conversation that we want to carry forward. And then actually you just drop basically all the messages and you just pass it its own summary to itself, to its future self. I've heard this called like a whiteboard approach where it gets to write on the whiteboard and then that's what gets passed. But I haven't— it's on my to-explore list.

Yeah, here you go. I think this is an excellent idea.

Let's go back to some upvoted questions. This is currently leading in upvotes, but I think it's quite hard and open-ended question. It is, how would you go about creating an agentic team of all of these models?

Yeah, I mean, you know, there are those multiple multi-agent frameworks, right, where you can describe. So, so I would basically probably start with, um, identifying the strengths and weaknesses of individual models, right? Because, you know, in this, um, probably like the, the easiest way to prototype and play with some ideas would be to leverage like one of the existing multi-agent frameworks. And, you know, in those frameworks, like typically you define agents, right? Those agents have descriptions so that you can put some routing or controller model in front of them. Based on those descriptions, the controller model would be able to redirect individual messages to the appropriate agent. I guess the good first step would be to identify what are the strengths and weaknesses of individual models. Once you have that, you can wrap them up with your APIs, right? But also, like, I guess, like, you need to be a little bit careful here because, you know, things are getting pretty hairy as the conversation is getting longer and longer. Like, if you just have, you know, an agent that is capable of certain things, but it is, like, caught, like, deep in the conversation and it may be, like, I don't know, missing some sort of context from, like, the previous exchanges, right? So, like, You need to make sure that you are building not only the proper routing layer but also the, as you said, like summarization layer, or like, you know, to make sure that you can like properly extract the context and pass it to that model and also make sure that, you know, that model is able to interpret this context correctly. But yeah, I mean, there are, I guess, there are like, you know, different approaches, and this is like the most most direct one because it's based on just like wrapping the existing models. But there has been also like a lot of interesting work around merging multiple models, right? There is this open source project MergeKit, and where you can, you know, take like layers from individual models and merge them together, creating those like Frankenstein type of models that composed of multiple pieces of existing models. I didn't personally explore this area, but it could be also a pretty interesting way to combine the skill sets of multiple models into one model.

I think that's a great answer. For what it's worth, when Harrison Chase was with us, he talked about he thought having multiple models together was a very exciting direction. And then I asked, like, is it really the case that, um, we think models have different strengths so much that you could actually benefit a lot from this routing? Um, he said that the thing that he's seen that's been successful is not so much, um, having different models with different strengths, but basically having like a big model and a small model. Then you're like, ah, this task, I just want it to be quick, And so we'll just use a small model. And here are some other tasks that's like strategic and complex. And we route it to the big model, which I hadn't heard that before. But that strikes me as less complex than some of the things I hear people talk about for like teams of different models that are all specialists in various ways.

Yeah, this is a great point. So it kind of depends what you are trying to optimize for, right? Like if you have no constraints, you know, you just go with GPT-4, right? Like, it's expensive, but it's intelligent, and, you know, it's also slow, but that may be okay. If your optimization objective is more around, like, the cost of latency reduction, then, you know, it definitely makes sense to start exploring those, like, more, you know, I don't know, hierarchical approaches where you maybe have some, like, intelligent model that doesn't require that much context, and, like, they're just, like, crafting this decision. And, you know, you basically get like a bunch of those, um, less, less intelligent but like more like specialized models. And I guess like, you know, function calling. So for instance, like, you know, what you could do is you could, you know, let's say fine-tune your model for like single-tenant function calling, right? And like if you identify that the use case, you know, fits within that, you just like use this model. And maybe like, you know, instead of using Llama 3, 70B, like you would use like way smaller model for this use case because it doesn't require as much intelligence and this way like you would reduce the cost. So I definitely buy this argument.

We've got, I think the next 3 questions are all questions I really like. So the next one from Andrew is, how does your fine-tuning compare to the Gorilla project from Berkeley's Bear? In terms of whether that's depth of function signatures, breadth of functions. He says, no, I've only briefly seen Gorilla, and he's not used— and he has not used it. But how does your work differ from the Gorilla project?

Yeah, so the Gorilla project is mainly, mainly focused on, um, single-tenant use cases and forced function calling and generating Python signatures, right? So typically there is an instruction from the user that results in function calling. And they actually support like a pretty good variety of function calling scenarios. So they have, you know, single function calling, they have parallel calling, they have nested calling. However, like based on my understanding, like the way that this model was tuned is tuned mainly for like pretty simple, like pretty basic functions that don't have like like too many parameters, or maybe like don't have like very complex parameters. But it is like, yeah, it is a model of a pretty, pretty decent quality. However, like, you know, as I said before, like many of the like real-world use cases that we see from our users like involve like very precise system prompting that like require a model to behave like in a very specific way, and also mixing conversations with function calling. So I think this is like, although I didn't like evaluate that model under those scenarios, but like, you know, for instance, the Gorilla leaderboard doesn't contain like any tasks for that. And even if you look at this slide right here, I specifically included the empty bench. And this is a multi-terrain bench, which basically doesn't involve any function calling. It is like the pure like, you know, just general instruction following, right? So I really wanted to include it in for our model to make sure that, like, you know, our model can still perform, like, non-function calling tasks. Nexus Raven is also, is also a very good model. It's actually, it includes some of the samples that, you know, include deeply nested, pretty complex parameters. So to some extent, like, at least if you look at the benchmarks, like, the Nexus benchmarks are more challenging than the Gorilla leaderboard benchmarks. If I remember correctly, like, that model is also focused mainly on the Python function calling. Yeah, I guess that's like the brief, like, summary of what I remember about those, you know, alternative models.

Okay, this next question is another one that I exceptionally like because it's someone who has like a real use case problem they're trying to solve, though it sounds extremely difficult. So I'll be interested in your reaction to this. Misha says, I have a very complex use case for Home Assistant. I need to train a model that will work with hundreds of functions, for a smart home and all AI assistant standard things, and the model should be local. Any idea on the smallest model that I can start with? And then there's more to it, but, uh, he says ideally I should be able to run without a GPU. Um, why don't we stop there? Does that, does that even sound possible?

I think it sounds, uh, possible. Possible. So I may like speculate a little bit and, you know, just play a little bit with those assumptions and see what we can come up with. So basically, if I were to support a very large number of functions and if I wanted to host the model locally, right? So in general, the inference speed will depend on the, of course, like, you know, the hardware, the number of functions which are going to affect the the context and, you know, also the size of the model, right? But there are some things you can, you can do. So, you know, for instance, what you could do, you could, you know, make sure that the function spec is provided, let's say, in the system prompt. So like it's at the top of the model and you basically don't change it throughout the conversation. So what you can do, you can basically, like, you know, even before you expose the model to the actual traffic, you run this system prompt through the model and, like, pre-populate the KV cache, right? And then, like, you know, this KV cache is going to stay in memory. So when you are, like, let's say you may have, like, multiple users chatting with the model, but, like, whenever a new session starts, you can just like, you know, copy-paste the KV cache that already has the definitions of the functions like pre-populated, right? So this is like one way in which we can reduce the serving cost. Another way would be, you know, we may, let's say, put those functions behind some sort of RAG, right? So like, you know, for each function, like we could basically compute like an embedding based on this function definition. And so we don't have to include them in the system prompt. And then, you know, based on like the conversation with the user, like you would, you know, embed the user prompt and match it with, let's say, top 5 functions and inject only those 5 functions into the prompt, which will reduce the size of the prompt. And of course, using smaller models could be an option. Those Phi models by Microsoft seem to be doing pretty well based on their size. So you could try them out. If 8 billion parameter model is an option, also trying the Llama family of models it could be an option. But yeah, so in general, you know, there are like many, many things you can, you can do here.

Yeah, yeah, he asked any idea of the smallest model that he can start with. So you said Phi, I think there's a new set of models out from like Quentoo also has like a 2 billion parameter model and should run without GPU. Okay. Um, very cool. I guess he said it has to be local. If you were putting that on an embedded device, then having RAG and all this stuff, it starts to get pretty complicated. But if it's embedded in on like a— or if it's local but on like a conventional system, then sounds more, more plausible. Um, I really like that, that RAG idea is a great idea. Uh, the next one I think is another really interesting question. Have you tried using function calling with GraphQL? Any tips or tricks, for example, by leveraging the graph data structure?

I guess you, you could. So in, you know, in general, like, it's a question like how to map like GraphQL to, let's say, function specs, right? And, you know, both are basically structured data formats. And like, you know, the function call, like, we call it function because like it is intended to basically invoke something, but like this, you know, something is a pretty abstract concept. So like at the end of the day, like, we're basically training the model to generate like structured data. And like GraphQL also like fits the definition of structured format. So I guess you could try, you know, existing function calling models and, you know, see if you, you know, define the graph as a function, if you will be able to like, you know, leverage the model to fill out the missing fields. Yeah, and also, like, you know, the grammar mode should be actually pretty useful in this case.

Cool.

Um, yeah, a lot of questions. Let me see what else has upvotes now. Um, okay, we've got a— I don't know how much time you have, but, uh, there's a question about whether you can demo, how to run the Fire Function demo. Wade says, I'm not sure how to provide it the function definitions to use.

Yeah, I'm sorry, I have an issue with the screen sharing, so I wouldn't be able to demo, but like maybe you can just like join our Discord and just like ping me there. We have a channel for function calling. Maybe I can follow up with you to share some, some examples.

Okay, and then is it this— is the way that you call it the same for Fire Function v1 and Fire Function v2?

Yeah, in terms of the API, we are just relying on the OpenAI compatible API. And so, you know, just, you know, go, go to the OpenAI SDK spec and basically follow the same structure of the input. Like, you can basically define the functions in the same way as you do through the OpenAI API and also like the same format.

Okay, but we should also join your Discord. I'm— maybe because I'm staring at these benchmarks, I'm like quite excited to try this stuff. So Hopefully others are as well, and we can— you have a bunch of us join your Discord. There's a question, it's from an hour ago, and it's, I think, a question about a very specific slide or point in the presentation. Is this called multi-agent? Is this called multi-agent? I'm trying to, I don't know if you remember some place where, uh, that question came up.

Second slide maybe.

All right, let's, let's see if we can go back. Probably this, uh, not this one, this one, right?

No, the Hub, just go to the second slide.

Okay.

The one, yeah, the, the, this one we mentioned multi-agent systems here.

Okay, commonly used to orchestrate agents in a multi-agent system. Uh, how do you define multi-agent system?

Um, so, you know, you can basically think about a function as agent, right? Um, I mean, you can wrap it in like any way, any way you want, but like this is like one way to to plug, you know, function calling into an SDK that, you know, has some abstractions that make it easy to build applications on top of. So there are many, like, open source, you know, multi-agent systems where you can basically, let's say, in the simplest case, define a Python function, right? And then say, okay, this is an agent. And, you know, the framework is going to extract the schema of this function, and that may be translated into like, you know, JSON schema object that can be passed into the function calling model. And then like when the model responds with a function call, like the framework can automatically parse the response and call the function, you know, get the response back and pass it into the model in the next iteration. But I guess like, you know, also this is a valid question in a sense that like I think the term agent is like abused quite a bit and is not like super well defined, so it kind of definition is context dependent. But I would basically, you know, to me, like an agent is like a component with like, you know, well-defined functionality that can be interacted with through a, you know, well-defined API.

Yeah. Um, we've got a couple of questions again from Wade about Thoughts on just using the same syntax we use to provide and retrieve results from OpenAI tool calling? I think you've said that you, that's what you do. Yeah. Cool. Okay, here's a question which I think comes back to some of the challenges with running fine-tuning systems in general. Any recommendations on how to handle changes or updates to APIs?

So, you know, in general, like, at the end of the day, fine-tuning is just a dataset and a process, right? Hence hyperparameters. So once you have prepared your dataset, you know, if there is any change in the APIs, like, I don't know, for instance, you know, we may want to go from Python syntax to JSON schema, right? Typically, like, what we do, we may store the data in some sort of, like, canonical format, like, either of those can be designated as the canonical format, and then we can basically have some sort of, like, you know, translation layer that, you know, during training when we read the data, we can, you know, convert them into different syntax. So I mean, this is one way, but also, like, you know, if we are standardizing around OpenAI APIs, like, you actually don't need to modify the model itself. Like, you can write some sort of wrapper around this API that is going to translate the call to the any, you know, syntax you, um, you need, right? And this is like, you know, typically what you do on the, like, on the, on the client side, right? Because you basically don't want to have like two context APIs. Like, you prefer to have like one API and then, you know, like different translation layers that the users can customize.

Yeah, and it seems like some people might fine-tune a model to make specific function calls And this would be a reason to say, actually, let's put the function definitions in our prompt and use a more general model so that we can more easily— when our functions change or our APIs change, we can just update the prompt rather than needing to train a new model potentially on data that is for the new API that doesn't exist yet.

Right. Yes. Um, okay.

Here's a question we get for, in different guises from many different presentations. I'll be interested in your reaction specifically for fine-tuning with functions. So question is, how much training data samples would be good enough for LoRA fine-tuning? So You could answer that in general, but I think we're especially interested in cases where you are fine-tuning for calling a specific set of functions.

Yeah, so actually surprisingly few samples. If you have high enough quality of data, I would say that maybe around 1,000, maybe even less samples could be good enough. So, you know, in many cases I have actually seen people investing time in hand-curating certain samples, right? Because this is like, you know, a few hundred samples is not something you cannot manage curating by hand. And, you know, also it kind of like, you know, depends what kind of techniques you use. Like, for instance, you know, whether you use like single-stage process where you just do supervised fine-tuning, or also if you use some sort of alignment with DPO, right? I would say for supervised fine-tuning, you need maybe around 1,000 samples if you do a LoRA tuning. Of course, it depends a lot on the base model use case and so forth. For DPO tuning, we have also seen very promising results with very few samples. Samples. Like, even having like around 100 samples should give you like reasonable boosts in the model quality.

Well, there's a question, another question here that I liked. What are best practices for generating synthetic datasets And here I'm going to add for function calling.

So I think the most important, so two most important things are that you need to have proper like prompt and seed datasets, but also having a good quality model to generate those samples is also super important. And of course, legally, you're not supposed to use the closed-source models, like both OpenAI and Anthropic and also Google. They basically won't allow you legally to use their models to generate data for tuning other models. So you have to be pretty creative. But on the other hand, if you use some sort of technique to also post-filter the samples because even if you use weaker models, on average, they are able to generate pretty high-quality datasets, but they kind of make more mistakes or maybe require a little bit more prompt engineering. So I would say that you will get a lot of false or bad quality samples with an open-source model compared to the closed-source model, but like if you do like proper filtering of those bad samples, then you should be able to get like, you know, comparable quality datasets. So like, you know, for instance, you know, one thing I can tell you from my own experience is that actually, you know, at Fireworks, we also have a family of multimodal models that we train on top of the LAVA architecture. And the reason why we did that was that the original LAVA dataset was generated with OpenAI models. And because of that, like those, like the original LAVA was not like permissive to use in commercial settings. So what we did, we kind of like regenerate the synthetic dataset that was used for LAVA using open source models. And then we retrained a version of LAVA. Based on this dataset, and we have seen like almost like no quality degradation, right? So it is definitely possible to use open source models for synthetic data generation, but like it would require a little bit more effort. And also like, you know, when you go into the synthetic data generation, also like have a plan. So I think it's more important to have like more variety of the cases as opposed to having more examples for the same case, right? So for instance, if you have this single term force function calling, you will get very diminishing results if you generate, let's say, more than, I don't know, let's say 200 samples for this case. So it's more important to have a plan. Okay, I want to support single term force calling, parallel calling, nested calling, and for each of those have, you know, a good prompt. And also, like, another thing that helps a lot, it may be obvious, is that including, like, few-shot samples in the prompt, it also helps a lot. So, you know, instead of just, like, providing the description of the data you want to generate, like, include a few samples of what you would like to see. That's helpful. Also, like, you know, varying the temperature helps because then the model gets, like, a little bit more creative But I mean, those are the pretty obvious things.

I have two follow-ups on that, so actually three. One is you said that you are doing some work with LLaVA. Do you have a model that can take both text and images as input and then do function calling?

And not yet.

Okay, when you do, I have Yeah, when you do, I want to hear about it. But okay, uh, the second— and I think there's some comments about this in, uh, the Discord— is some people, uh, I think that you mentioned DPO in the context of function calling. How does that work? Like, where do the preferences of what's better and what's worse come from?

Right, so like typically, you know, I'm not sure if it's a typical approach or not, but like We would do the supervised fine-tuning first and then let's say share the model with our Discord users and they would start reporting some failure cases. I see. And some of those failure cases could be translated into more generic DPO negatives because the difference between SFT and DPO is that basically DPO has also negatives. We not only teach the model what to do, but also we tell it what not to do. And especially, I don't think that DPO is that useful if you have very simple system prompts, but DPO gets really useful if you want to make your model follow the system prompts more precisely and The reason kind of goes back to what I mentioned before is that you have typically very limited amount of instruction data with complex system prompts. And DPO is kind of a little bit more efficient in terms of utilizing the information, right? Because you have to show a lot of positive examples for the model to discover some pattern of what to do, but it takes relatively fewer samples if you can mix like both positives and negatives. So show it, okay, do this, but like, don't, uh, don't do that.

Yep. Um, and, and staying with this idea of distinguishing better from worse, when you generate synthetic example— synthetic, uh, data, do you ever generate both the code and then some tests that you want to run on it and then use that to filter out the bad code that would then go into your SFT step?

I mean, that's a great idea. We actually don't do it, but I think it's a good idea.

I see. Okay. What else do we got here? This is a question. From a while ago. You don't have— you can choose to take this seriously or not. How does it feel to secretly play with AGI?

I mean, it feels good, but I also don't think about it this way, right? So, you know, you have a problem at hand and you want to address it. Like, you basically want to fix it. You know, when you run the benchmarks and then like, you know, you can see that, you know, your model is behaving like well in the scenarios that you didn't like even think about before, like when you were preparing the data, it's like definitely like, you know, very rewarding because then you can see that the model was able to like generalize kind of like beyond what like we kind of envisioned for it. But, you know, I still feel that, you know, we are quite far from like the general like AGI capabilities, but like definitely like, you know, function calling will make it more real, right? In a sense that, you know, if we do have like AGI capable model and then through function calling, we basically give it access to the real world, right? So I guess like, you know, if you are like, you know, very pessimistic about the future of the, of the human race. And, you know, you basically give them all the tools to, uh, to do things in the real world. And like, the function calling becomes the, the proxy to, uh, to that is kind of like an interesting, um, you know, thing to think about. But like, I, I don't think we are very close to that. So I basically don't honestly think too, too much about it at this point.

Yeah. Um, well, the most recent question was another one about AGI, so I'm gonna— I think given your last answer, I'll skip that one. Um, yeah, we've got a lot of, uh, we're also coming up on I don't even know whether it's 2 times or 3 times the original scheduled time. So let's see what else we got here. Yeah, I think this is a question that I asked, and whether there are just best practices for creating custom function calling datasets. I don't know if you have anything else to add beyond what you've said.

Well, it is, you know, it is definitely important to have the right data. Actually, maybe I can answer like a different question. So like people are also interested in like how important is the data versus hyperparameters, like versus the base model, right? So I would say that, you know, The recent trend we have seen, at least with the fine-tuning, is that since the models are getting more intelligent, people are using less data, then the quality of the data becomes more and more important because you want to maximize the signal density in your data given that you have limited number of samples. But we have also seen that if you train with fewer data samples, then like the model also becomes more sensitive to hyperparameters. Uh, so you know, every single like, like, you know, 2x or like 3x change in the learning rate can make a big difference on the, on the quality of the model. So like, you know, it definitely makes sense to, to play with the hyperparameters when doing fine-tuning. And also, even with the same learning rate, it may make sense to run a few training runs. Unless you fix the random seed across the entire board, there is still some randomness that leaks into the tuning process. And you may actually get slightly different quality models, even with the same datasets. So it helps to have multiple runs and, I don't know, pick the best variant. So yeah, as I said before, if you don't have to go into fine-tuning, just don't go there because slides are one thing and it may sound easy, but at the end of the day, if the use case is reasonably complex, it will definitely take like a lot of time to not only like prepare the right data, but also like, you know, to play with different kinds of parameters and like do tons of tests and, and so forth. So it's not super easy. It is rewarding, but like it takes time.

And therefore, this has been— this is one of the last talks of our conference. This has been so, so interesting or enjoyable. For people who want to follow you and like keep hearing about all the new things you're doing, what's the best way to keep track of, um, your work?

Yeah, sure. So, uh, I'm not as active on Twitter or X as I should be, but I have an account there, so you can, you can look, look me up. Uh, I'm like a little bit more active on LinkedIn, so yeah, feel free to look me up there. Also, just like, you know, follow Fireworks HQ on X and on LinkedIn. And we have Discord where we are also pretty active, so, you know, feel free to join it. We have a channel for function calling. Yeah, and thank you so much for having me. It was like a very enjoyable experience to hear the questions from the community. And yeah, I'm basically looking forward to getting more feedback and working with you guys. Yeah, well, as a company, we're really committed to supporting developers and that's why we are investing in basically cheap and efficient inference, right? Like our inference services like Builder and LoRaTuning that you can and used to, you know, basically experiment with like different variants. Like the model hosting is, is pretty cheap. Um, so, you know, we are like, you know, very motivated to make the developers happy and always, you know, interested in, in your feedback. So definitely keep in touch.

Okay, um, I will do that, and I think many of us will. Um, and it sounds like actually getting into the Fireworks, um, uh, Discord might be a good way to just stay in the loop about what you guys have going on as well.

Yeah, yeah, we are— as a company, we are active on, on X. Like, it's just like me personally, I'm not, I'm not super active, but yeah, definitely follow us on, on X and on LinkedIn as well.

Okay, sounds good. Well, thanks so much.

Yeah, thank you for having me. Yeah, have a good day.

Bye.