Gemini Flash is SURPRISINGLY Good for Agents and Function Calling

The Gemini models recently had an update to both Pro and the Flash version and with this update now you have access to improved rate limits and soon you will be able to fine tune the Flash version on your own data set. They also improved the JSON mode and hence function calling that is going to be the focus of our video now. This update does seem to have improved the performance, so with the recently released ranking of Elemsys Chatbot Arena Leaderboard, both the Pro and Advanced version of Gemini are sitting at number two while the smaller Gemini Flash is at number nine, just behind GPT4 and Claude Opus, which is pretty impressive. I'm personally interested in Gemini Flash because it is sitting at a sweet spot when it comes to quality of outputs, price and throughput. So here if you are interested in a good quality model at increased throughput or tokens per second, Gemini Flash I think is a really good candidate for that. This seems to be better than something like Cloud Haiku but can provide increased throughput. Similarly, if you're looking for a compromise between the quality and price, again, Gemini Flash is better than haiku and GPT 3.5 but the cost is lower compared to both these models. Now when it comes to LLMs, my main two use cases are rag and Agent or tool usage. So I wanted to look at a practical use case of Customer Support Agent which is going to be making both sequential as well as parallel function calls. But before that let's understand what exactly function calling is and when would you need that? Let's say you want to have access to the real time information, so for example stock prices or weather information and the LLM is not able to provide you that information because it's not in their training data. So normally what people do is they will use external APIs and then give the LLM the ability to pick one of these APIs which are wrapped around a function to interact with real world, hence function calling. So here is how the flow looks like the user query comes in. The LLM will look at all the available tools or functions and it will determine in the first step whether it want to use an external function or not. If it decides not to use a function, the LLM will generate a response based on its training data. Now if it decides to use an external function, first it needs to pick the appropriate functional tool from all the available functions. Now keep in mind that the LLM itself is not able to execute your functions. Those are going to be executed by you or using a code interpreter. So the LLM will only provide the inputs Based on the initial user query plus what the expected outputs are, you will have to execute that function call, get the response, and then that response along with the original query is fed back into the LLM to generate a final response.

For this video tutorial, we're going to be building a customer support agent that is going to have the ability to bold do sequential as well as parallel function calls. We will start off with a simple setup and we'll add complexity throughout this tutorial the way Gemini does. Function calls are very different than the other proprietary LLMs frameworks that I have seen and we're going to look at the differences. Now, to get started, we will first need to install the Google Generative AI Python package. Then we need to import different packages that we'll need throughout this tutorial. After that we'll need to set up our API key to interact with Gemini Flash. You can get your API key from Google AI Studio, so go to your account. Then go to API keys, click on Create API key and copy that API key. If you're using this within a colab, you will need to set that API key as a secret and make sure to enable access to that specific API key for your notebook. Now, if you're using this locally, you will need to set that as an environment variable. So this function block will be able to handle both situations. Next, let's look at our chatbot. So to begin with, we will be able to only do two operations using this agent. One is going to be get order status and the other one is going to be initiate returns. Later on we'll add more and more operations that the agent will be able to perform. So to get the order status we need the order number and there is a dummy order data set. This is basically a dictionary which has three different orders and three different statuses. But in reality you will be probably implementing another API call which will be getting this data from an external database. For initiate return we get order number and then a reason of why that request was initiated and it will just return a string. Now all the functions are going to be implemented like this. One of the most important aspect is this doc string which actually is going to tell Gemini Flash what a specific function does. So whenever you're implementing your own functions, make sure to pay close attention to this description. Okay, so let's set up our client. We're going to be using the Generative AI package. Then we'll use the Generative AI model. We'll need to provide our model name. So I'm using Gemini 1.5 flash and we're using the latest version, then you will need to provide a set of tools. In the beginning we're going to only use two tools and those are the get order status and initiate return. Now you can have a look at the list of tools that are available to the model by using the tools and then to proto. So here you're going to get two tools. One is the get out of status, a corresponding description, then the initiate return and the corresponding description. And it also tells you what is the input, what is the type of input. Now, in order to use function calling, you will need to use the chat version of Gemini. So for that you will need to start a chat session. And the way you do it is you're going to call the chat start underscore chat function on the model object. Now, if you want this model to be able to also do function calls or execution, then you can set this enable automatic function calling key to true. And this will not only take the user input product will actually execute the whole loop that we are talking about. So using that single line of code, you will be able to execute this whole loop, which is very different than some other platforms like Mistral, OpenAI or Claude. And if you don't want Gemini to do the actual function calling, you can do that yourself. I'll show you later in the video and on how to do that. So basically we create a chat object and then we're going to send message to that chat object. So for example, this messages what is the status of order 1, 2, 3, 4, 5. And the response that we get is the order is shipped. Now if we look at the actual order status, it indeed is shipped. So it's a very simple function call that we are making and it seems to be able to get the status of the order. So this is a pretty good sign. So in order to understand what exactly is happening, we're going to look at the chat history, which are the internal communication that are happening between the model and the user, right? So essentially we are going to look at all the previous communications or conversations.

So the first user input is what.

Is the status of this order. Then the model basically determines that I actually need to use a function for this. And based on the description of the function plus the actual query, it decides to use get order status function. And the input is going to be order ID that is coming from the user input. Now the second step is to actually execute that function. But the way Gemini does this internally is it expects the user to execute that function and that's why you see user. So basically it pass on the order ID to that function and it will get a response. So the Python interpreter actually executes that function as a user and that is passed on to the model and the model is able to generate our final response. Okay, so let's look at another example in which we want to return a specific order number because it's defective. So it says, okay, I have initiated a return for that order. You can also look at the candidate responses. And this one is especially useful if the model has to generate sequential or parallel function calls. So we'll look at an example later in the video, but there are going to be multiple parts. If you have parallel function calls, we.

Can again look at the internal working.

By looking at the chart history. So since we are using the same object, we are going to see the previous chat conversation that happened. But now the user asks a question, the model determines that I need to use the initiate return function. Then again it will use the user role to actually make that function call, and that output is going to be passed on to the LLM to generate a final response. Okay, now let's look at sequential or nested function calls. In this case, the model has to make multiple file calls in which the output of the second stage is going to be dependent on the first stage. So just to have a quick review, here are the orders with different statuses. And I said, can you check the status of order this if it's delivered, please initiate return as it was at the wrong order, right? Now it says the order has been delivered. I have initiated a return because it was wrong order, right? And we can actually look at like how exactly does it do that? So it looks at the order status, it is delivered, and after that it says that okay, it is delivered. So I'm going to initiate a return, right? And again we have that conversation that happens between the user and the model. Now let's say what happens if the order is actually not delivered. And we say, please initiate a return because it was a wrong order. Now in this case the order number is 67890, which is still under processing. So the model response says, this order is currently being processed. I can't initiate a return until it's delivered. Now let's look at a more complex case. So we have the same prompt, but at the end I said else cancel the order, right? So either initiate return or cancel the order. Now the response is the order is currently being processed. I can't initiate a return until it's delivered. I will try to cancel it for you. But the problem is that there's no way the model will be able to actually cancel the order because there is no function associated to order cancellation. Okay, so to give the model the ability to actually cancel the order, we are going to add the cancel order function. We'll update the list of tools. Now we create another chat session. So basically we discard everything that we had before we run the same query through the function or through the model again. And in this case the response is okay, I have canceled the order. Now let's look at the step by step process of what exactly happened here. So based on the initial prompt, it actually looks for the status of the order and it's able to figure out that we are still processing it so it doesn't have to initiate the return. In the next step, it's going to cancel the order. So for that it makes a call to or it picks the cancel order function. Then it will use the user role to actually make a call to their function and the final response is going to be that I have canceled the order. Right. So as you can see, this model is actually pretty good because it can do these multi stage nested function calls and the responses seems to be accurate, at least with this specific case. Now for my quick experiment, it's actually even able to handle relatively complex prompts. So for example, here we are trying to ask it for a status of one order, then ordering it to cancel another order and initiate return for another order, because that is defective, right? And here they are the actual statuses. So based on the model response, it says the status of order, this is shipped because that's the order the the status that we wanted and it's actually shipped. Then it's able to cancel the order for us and initiate return for that other order. So it can do multiple calls sequentially or in parallel depending on the situation. And this is actually pretty impressive. Now how many functions can handle. So I actually increased the number of different functions to a total of 10. OpenAI I think recommend to keep them.

Under 200 for their models.

But for this example we just created 10 different functions which looks at different aspects of a customer agent. So you have the first three orders function similar to what we had. Then you can update address, you can track shipments, apply discounts, and so on and so forth. Now for this example I'm going to show you how to execute the function calls yourself and it's going to be making parallel function calls. So again, similar to the previous case, we will create a model object. But now the list of tools is pretty huge and we're going to start a chat model. Now you will see that I'm not passing on that automatic function calling flag. So the model itself will not be able to make function calls anymore. It will return a list of function that it thinks needs to be used and we will have to make those function calls ourselves. So I wanted to show you this example as well so that you have a much better holistic overview of how this process works. Now, even the function call is a little more complicated. So it says what is the status of order? This can you update the address to this address? Right. So now it has to first figure out what or actually make two parallel function call, because these are completely independent things, although both of them are reliant on this specific order. Now, if you look at the response candidates, so it has actually two parts. Now the first part is get order status. So this is basically the first part of our query. And in the second part it's going to be using the update shipping address, Right? So it can call these two function in parallel if need be. Right? And that's why you have two parts in the response now. So what I'm going to do is I'm going to get the actual functions out of those two parts and execute them myself. Right?

So we basically run through this loop.

We get the actual functions, then what are going to be the input to that function. So for the first function, since we are just getting the order status, we will pass on the order number. And in the second case we are going to update the shipping address. So we need to pass on the order number plus the new address. The output is just the list of functions plus the corresponding inputs.

Now if I want to make those.

Function calls myself, so I'm going to be using this dictionary. I have a dictionary get status order, but I actually make that function call by simply providing the order number to get the status. And the second one is to get the address updated. Right? So here's how the function call for the first one is going to look like if I had to do it. If you provide the custom or the order id, all you get is the status. Right? And that's exactly what we want. Right? And in the second case we are going to get the updated address. Now, in order to get the final response, we will need to provide both.

These functions and their results just like.

The way I was showing you in the original loop. So basically we are doing this step. The model picked the appropriate function, we executed the function calls Then we give the output of those function calls to the model again to generate a final response and with that you will be able to get a correct answer. So it says the order is currently shipped. I have updated the shipping address to this address that you provided.

Now overall I think it did a pretty good job.

So out of those 10 different functions it did not really confuse one function with the other one, which is pretty impressive for the model, which has pretty good speed and is relatively cheaper compared to other models in its class. I hope you found this video useful.

Thanks for watching and as always, see.

You in the next one.