Claude 3 Introduces Function Calling and Tool Usage

Anthropic just introduced the ability to do function calling or use external tools with Claude 3 family of models. In this video, we're going to look at what tools actually are, how do they work, how do you define your own tools? What are some best practices around tool usage and function calling? And at the end we're going to look at a practical example of how function calling works with Claud 3. But first let's understand why you need function calling or external tool usage. So by their nature, the LLMs have certain limitations. For example, Most of the LLMs that we have seen are not good at doing mathematics or calculations. Or each LLM has a certain cutoff training date beyond which it's not going to have any access to the data. So you probably need to do a web search to augment the LLM when it's generating responses. So for that you give the LLM the ability to use these external tools or do function calling to specific function which implements this functionality. So with function calling, here is how the flow works. Whenever there is a query from the user, the LLM will look at the available tools and it will decide whether it needs to use a tool or not. If it doesn't need a tool, then it will use its internal training data to generate a response. Now if it determines that it needs to use a tool, for example, if it has to do certain calculations and it decides that I need a calculator, then the first phase is that among the available tools it needs to determine which tool to use. So let's say if it was supposed to do calculation, it will pick a calculator. If it's looking for weather condition, it probably is going to pick a web search. Right? So first it needs to determine which tool to use and then it needs to make a call to that tool. So it could be a simple API or an external function that implements the tool usage and then it will get a response from this function or the API call that is going to be sent to the LLM as a context along with the query and the LLM is going to generate a response. Okay, so this is how the general function calling flow works. Now let's look at specifics of Claude 3 family. Okay, so let's look at how you define tools. So this functionality is in a beta. That's why whenever you use the anthropic client, we're going to be using a beta rather than the normal client. But I think it's a pretty mature at the moment. Now let's look at the tool definition. There are two components. One is this description based on which the cloud will decide which tool to use. And the second version is the actual implementation which can be an external API or an external function. And we are going to look at that in more details when we are looking at the example. So each tool needs to have a name, then a very detailed description. You want to make sure to provide as much details as possible in the description. Because this description is used by Claude family to determine which tool to use. Then there is an input schema. So this basically determines what inputs are going to be required from the user. So let's look at this example. In this case the user query is what is the weather like in San Francisco? So based on this query, if Claude decides to use this tool, which is the getweather, then it needs to figure out location as a required field. So the location we already know from the user query and that location is going to be then passed on to the actual external API or the function which will return the weather for that location. Now this will become a lot clearer when we look at an example, but in terms of the best practices, you want to make sure the description is highly detailed. So you need to provide details regarding what the tool does, when it should be used. If there are multiple tools, what each parameter means and how it affects the tool behavior and and any important caveats or limitations such as what information the tool does not return if the tool name is unclear. This is extremely important because based on the description alone, Cloud will decide which tool to use. Okay, so let's look at an example. This is based on the cookbook provided by the Anthropic team. So we're going to be building a customer service agent with client side tools. So this is basically a customer service chatbot using Cloud3. And this chatbot will have the functionality to search for customer data, pull up details of orders, and even cancel orders for the customers. So this will need three different tools. One is search for customer data, then we need another tool for pulling up details of the orders, and then another tool that has the ability to cancel orders for the customer. Now to get started, we're going to install the Anthropic package. This is the Python client. Next we need to set up the Anthropic API key. Since I'm using Google Colab Notebook, I set this up as a secret here. But if you are using this on your own machine, you can just set this up directly. I'll put a link to this notebook in the video description. Next we need to set which model we want to use. So for this specific example, I'm using the Claude 3 opus. However, this will also work with the Claude 3 haiku as well. If you have simpler tools that you want to use, I think Haiku is good enough. For more complex scenario, you want to use Opus. Specifically if there are multiple tools that need to be called in certain order. So for example, if there is a serialized calling or chaining of tools, then you definitely want to use Opus. Okay, so let's look at the client side tools or details. As I said, there are three different tools. One is we're going to use to get customer info. The second tool is to get order details, and the third tool is to cancel orders. Now for each of the tools you need to provide detailed description. So for example, that's the first one, this is the second one which retrieves the details of a specific order based on the order id. Right? So the first tool needs customer ID as an input. The second tool needs the order ID as an input and the third tool also need customer sorry, the order ID because we're using this to cancel orders. So based on the user input, the model has to decide which tool to use and that is based on the description in here. However, the actual usage is going to be separate. That usage is implemented through external functions. So we have a get customer info function, then get order details function and then cancel order. And these have their own implementations. So based on the first step, the LLM will decide which tool to use and then it has to make a call to the specific tool. Right? So you need both of those parts. First is the definition of tools and then the second is actual implementation of the tool usage. Okay, so now the once the LLM decide which tool to use, we are going to simply get this function name right? And then we use this function to do the actual function calling. So for example, in the LLM decides that it needs the customer info. So we will call the get customer info function if we need the order details. So we'll call this get order details function. Or if we want to cancel the order. So we're going to call this function. Okay, so this implements the main loop of how the function calling works. So I'll just quickly walk you through this. So first we get a message from the user. Then since we want to use external function, so we are using this beta tools messages create. Basically we pass on the user message to the LLM right here. It needs to determine whether it will use an external tool or not. Right. Now if it decides to use a Tool, then it will stop the response. And the reason for response stoppage is going to be tool underscore use, right? So let's say it decides to use an external tool. Then it kind of goes through this whole cycle of determining which tool to use. Then once the use the tool is selected, that is passed on to this process tool call, which is basically the function that we defined in here, right? That will give us the results. That result is dumped into a JSON, passed on to the LLM again, right? Along with the initial user input or query, plus the details of the response from the tool or the function calling, right? And the model will generate another response for us. So basically it implements this whole loop where it decides to use an external tool, pick up the tool, make a function call, get a response, feed that back into the LLM along with the initial query and get a response. Now, in order to test it, so here was a query. Can you tell me the email address for the customer? C1, right? So here is basically all the steps that the LLM has to take. So first it determines that it wants to use an external tool. That's why the stop reason is use external tool. And it determines that to retrieve the customer's email address, the get customer function should be used, right? So it makes a call to that function, gets the return, like gets the results. Those results are feeding back to the LLM along with the initial query. And based on that, you get a response. Now, similarly, if you say what is the status of order 2? Then it has to go through a very similar steps. And then here it determines that in order to get the status of order, it has to use the get order details function, right? It will get the response, and based on the response, it will feed that into the LLM again, right? And here's the final answer that we get from the LLM, right? Similarly, there's a case of please cancel order one, right? So it will follow exactly the same steps. Now, let's say if I say write a Python program that prints numbers 0 to 100, since in this case it doesn't need to use an external tool. So the stop reason is end of the turn, because once it evaluates this query from the user, it will determine that I don't really need to use any tool, so it's not going to have stop reason as tool usage, right? And here's the kind of response you can see. It says the user request is to write a Python program that prints the number is 0 to 100. This is a straightforward programming task that does not require any usage of tools, right? So that's why it will completely ignore that and it will write the response or the program. Now in this case, it will simply write the program itself and will show the results to the user. Now you can do exactly the same thing with Haiku as well. So I just repeated the same prompt or query. With Haiku instead of Opus, we get the same response. Similarly, we can check the tool usage. So this was the same prompt that we used before, but this time the model is Haiku and we get exactly the same response that that we got from Opus. Okay, so this was a quick video on how to start working with tool usage or function calling within Tropic. However, you can do a lot more complex things. For example, you can have sequential tool usage, right? We're going to look at some more advanced concepts in upcoming videos, so if you're interested in that, make sure to subscribe to the channel. Thanks for watching and as always, see you in the next one.