Taking Function Calling to the NEXT Level with Groq API 🚀 🚀 🚀

Today we're going to look at the fastest function calling that's possible within LLM. For this we're going to be using the Groq API. Thanks to the open source community, there was already support for function calling, but that was unofficial. But now the Groq team has implemented official support for function calling or tool usage, which is pretty great. Currently it supports three different models, so Llama 270 bill, the Mixtral Moe and Gemma 7 bill. Function calling enables the model to interact with external world by making API calls. So here are some specific use cases that you can use this for. Convert natural language into API calls, call external APIs, for example if you want to get weather information, or do web search or resume parsing for recruitment. These are different examples use cases that you can do through function calling. Now we will walk through a code example and I'll explain this step by step. But here are basically the steps of sequences that you will perform. So you will initialize the API client. This is going to be the Groq client. Then you need to define the function and conversation parameters. So these are different parameters that you're going to be passing on for the model to make requests. Then process the model requests. You want to make sure that the model evaluates your prompt. And then based on that it will determine whether it needs to use external tools or not. And that's why it's going to be using or making these function calls. And then once it gets a response from those functions, then incorporate function responses into conversation. So basically it's going to be creating two calls to the LLM. I explained this in my previous video. I'll put a link to that video, but here is how the flow looks like. User have a query. Then the LLM determines whether it needs to use a function or not. If it doesn't need to use a function, it will just generate the response. If it needs to use the function, then it will pick appropriate tool to use, then get a response from the tool and pass that response to the LLM along with the initial query and the user will get a final response. Okay, so let's look at a code example. This is based on the official example that is provided Groq documentation. So first we need to install the Groq Python client. After installing that, we need to set up our API key. In order to get the Groq API key, just go to Groq Cloud, click on API key, then create another API key and copy that. Now, since I'm using Google Colab Notebook, I am using The Secrets API in the notebook and just making this specific secret available to my notebook. So basically I will use the user data get function to get the API key and create another environment variable. After that we import the Groq client since the output is going to be a structured output. So we also need to import JSON. For now, for this specific example, I'm going to be using the Mixtral Moe model, but the same will work with llama270. Bill, I haven't tested the performance on Gemma since it's a smaller model, so it might have some issues with function calling. Okay, so let's look at an example of what exactly we are doing here. So here the objective is to get the current score for a given NBA game. So basically the user will provide name of the team and then depending on the name of the team we want to return a certain stats. So if you look here, for example Warrior had a game against Lakers and then it was an away team. We have the final score as well. Right. So think about this is external information that you need which is not included in the training data of the LLM. So this could be web search or looking for some financial information, but in this case we just want to get information regarding an NBA game. So this is the function that the LLM is supposed to call, but the LLM has to make a decision on which on when to call this function. Okay, so here's the main code. Step number one is send the conversation and available function to the model. So there is going to be a system role. This is the system message that we are passing on to the LLM. You are a function calling LLM that uses the data extracted from the getgamescore function to answer the questions around NBA game scores. And we also want the function to include the team and their opponents in the response. Then we get the prompt from the user. Now let's look at tools. So in this specific example we are using a single tool, but you can include multiple tools and then the LLM will have to make a decision which tool to use depending on the user query. So first you need to define the type of tool. So Grox recommends to use function type, then the name of the function, then a detailed description of what it's supposed to do if there are any parameters that you want to pass on. So you want to provide a list of parameters in here, one is the name of the team and it's a specific description. So you can have multiple parameters that you're passing on. Each of those is going to have the name, type, description, and what is the required output from the function call. Okay, so after this setup, we're going to make the initial call to the LLM. In this case, we're going to be using the Mixtral Moe. We pass on this message based on our flow diagram. The LLM has to determine whether it wants to make a function call or not. So it will generate an initial response. Then we're going to check whether it wants to use a tool or not. So that is basically the function calling part. If the LLM determines that it does not need to use a tool, then we will just return the initial response from the LLM as our final response. But in case, if the LLM determines that it needs to use a tool which is coming from this response.response message. Tool calls, then basically we look at all the available tools. Then the function, the model will select which tool to use. Depending on the user input, we get the response from that function, and that response is appended to the original output from the LLM and that is passed on to the second call to the LLM. So basically we're talking about this flow. So if there was a tool usage, then it will pick the appropriate tool, make a call to that, get a response, and then feed back into the LLM and we return that response to the user as our final response. Okay, so let's look at an example. The first one is what is the score of the warriors game? Now, the initial call to the LLM will determine whether it needs to call a function or not or use an external tool. So here's the response in this case, if you see it actually determined that it needs to call that function getgamescore. And the team name is going to be Golden State Warriors. That is based on the user input, right? And it will make the second call to the LLM based on the output of the function. So the final response is this. So it picked the score. And it also told us about the opponent team as well. So this seems to be working absolutely fine. Now, what happens if we ask a question that does not need a function call or use of an external tool? So in this case, the prompt is, what is the purpose of life? The initial response from the model was, I'm here to assist with the tools and functions you have provided which pertain to making calls for NBA game scores. Since those tools are not applicable to answering the purpose of life, I cannot use them to provide factual answers. And then it says, however, the purpose of life can be interpreted and answered in many different ways. Right. So based on the initial response, the model determines that it does not need to use any of the tools that we have listed and it will just give us a response from its own trained knowledge. Okay, so this was a quick example. Now, as I said before, you can actually expand the list of tools so it can be more than one tool. And in that case you will need to just expand the list of available functions in here. And you will also need to modify this section as well, because we want to be looking at multiple different tools. So the key that you're going to be using is going to be different. Let me know if that's something you would be interested in for me making it a video. But the responses that you get from this are extremely fast and that's one of the main selling points of Groq API. Now, the API from Groq is still free to use, so I'll highly recommend to experiment with it if you're interested in the paid account. I think they are putting that together. Will be interesting to see when that comes in and hopefully they are going to have much better rate limits. In that case, I hope you found this video useful. Thanks for watching and as always, see you in the next one.