Create Anything with LLAMA 3.1 Agents - Powered by Groq API

One of the capabilities of llama 3.1 that meta highlighted in the release is function calling or tool usage. I want to put it to a test. So this video is not going to be a tutorial on how to do function calling with Llama 3.1, but I really want to see how further we can push it. Meta released an Agentix system around function calling which I have yet to set up locally. So in this video we are going to be relying on the Groq API because it provides one of the fastest API for interacting with Llama 3.1. We're going to test both the Llama 3.1. 70 billion as well as the 8 billion model. Unfortunately 405B is not available through their API yet. Groq also has a version of llama 70 billion that was specifically fine tuned for function calling, so I'm going to also compare that and see how good this is at parallel and nested function calls. We're going to also look at the observability aspect of function calling and tool usage by LLMs. For that we're going to be using Lang Trace, which is an open source and open telemetry observability for LLM applications. More on that later in the video. Let's look at the code. There are going to be several notebooks that I'm going to share in the video description and each one of them is going to be using different first we need to install all the required packages. This include Langtrace, Python SDK, the GROQ Python SDK and OpenAI Python SDK. We don't need OpenAI since we're not using the LLM, but this is one of the dependencies of Langtrace SDK. Next we set up our API keys for this experiment. We don't really need langtrace, it's ability platform, so it can track the the number of requests and number of tokens that are being communicated between your local environment and the LLM API. So it's great for that purposes, but the code will work without langtrace as well. However, if you want to understand the usage of your tokens, Langtrace is a really good option. It's very similar in functionality to LankSmith, which is an observability platform from LangChain, but it has support for a number of different vendors so you can directly use it with OpenAI croc, cohere perplexity, or you can also use it with LangChain and Llama Index. It's a relatively new project, it has a Cloud hosted version or if you want, you can self host this as well. So for this video I'm going to be using the cloud version. You will need to make an account once you sign up, then create a new project. So I'm going to call this Agent 2. Now you can see that I already have one project, but I'm just going to set this up. We will need to generate our API key. I copy my API key and create another secret in here which I'm calling Langtrace API key and also make sure that it's accessible to my notebook. Okay, so when you run this you're going to see a number of errors because I have not installed these different packages. The only package that we are tracking is the Gro LLM API. So we should be able to see traces from the Groq API. Now as I said, Groq is providing a number of different. There are two options currently available for llama 3.1. One is the 70B and the other one is the 8B. I'm going to be testing both of them. We're going to start with the bigger model. We will start with a very simple example. So there is going to be a single function that the model has to call. Later on we're going to add a lot more functions and then test the abilities of this LLM to do multiple parallel function calls as well as nested sequential function calls. Those are going to be really good tests for testing the ability of this this model to be a really good agent or not. So the function that I'm using is getgamescores. It's supposed to receive a team name and then I have generated dummy data that basically shows which team it's playing against, what were the scores of the game, and the model needs to determine which team won the game. So this first part is going to be just to show you how the function calling actually works. If you're using the Groq API and you probably have seen this flowchart if you have watched my previous video on function calling or agents. So essentially the goal is for the LLM to pick an appropriate function or API that it can use for a specific task that is in the user query. Now, the main thing is that it can only pick the function or tool to use and also generate the appropriate inputs, but it cannot do the execution or can call that function. For that, you need to use a Python interpreter to do that function call, then return the results of that function call to the LLM again to generate final responses to implement that Loop we're going to be using this function called runconversation, which gets the user input that's basically the query. Then it has a specialized system message or role. Now for this specific example we are making it very easy for the model because we are telling it you are a function calling LLM that uses the data extractor from the get underscore game score. So it knows which function to use. Right. Later on we're going to make it more general. That is going to be more appropriate for agents. So it has to use that function to answer questions around the NBA game scores. Include the team and their opponents in your response. Right? So we're going to be only providing the team. But from the function that we provided or this definition that we have, it has to also figure out what are the opponents. We take this system role along with the user query to create this messages list. Then similar to the OpenAI structure, we need to provide a list of tools. So here we're defining that the tool that we're providing is going to be a function. What is going to be the name of the function? A small description and then what are the input and outputs. So in this case the function expects one input which is required is the name of a team. We take that, we use the model that we selected, which is 70B3.1. In this case they take the messages, we provide the list of tools and then it has to automatically make a choice of which tool to use. And we are also limiting the number of tokens that it can generate. So basically here it's going to make a decision whether to use a function call or not and then select the appropriate function plus provide the inputs. So first you are going to get a response from the initial call to the LLM. Then GROK uses a special variable called toolcalls. So you need to check if a tool was used during the initial response or not. If the tool was used, then you get the name of the tool. In this case there is only one, which is getgamescores. So we will need to get the list of all the tools that were used, get the inputs from the model that it will generate. I'll show you how it does that and then add that to the messages and make another call to the LLM. This is going to be this loop and the LLM is going to generate our final response. Okay, so here is a simple prompt or query. What was this? Game OF THE Warrior Games now you can see here that how the internal mechanism works. So this is the first call in which it determines that. Okay, we're looking for Golden State warriors and it is supposed to use the getgamescore function. Then it passed this information on to the second pass of the LLM. Right? The function will generate a response. It will use the output from the function to give us the final response, which is that the Golden State warriors played against the Los Angeles Lakers. Here's the score. Now, it actually got the correct responses in this case. So the functional calling seems to be working. Let's go to langtrace and see what exactly happened in here. So there were two calls to the GROQ API endpoint. The first one was the original system message plus the user input. It took about 90 input tokens. The output tokens were 77. So here's the total number of tokens that were generated by the LLM. Right? Then based on the output, which is here, this is basically what the function call returned. It generated another input, which is this. And then the output based on that is this one, the final response from the model. Right. And again you can see that it took about 200 tokens in total for the second function call. The Langtrace platform has a lot of different other things that you can test or observe. Right. I'm going to go through them in another video. Also, as a free user, you get about 50,000 free traces per month, which is pretty generous. And here you can actually see the number of tokens that are being exchanged. Now, since KLOC doesn't charge you anything, it's showing the total cost to be zero. I think for OpenAI LLM, it will actually show you the cost associated with the LLM API calls. Okay, so this was a very simple example to get you started. Now let's look at more complicated example. In this case, we are going to be testing its ability to do parallel tool usage or parallel function calls. So we are expanding the number of functions that are going to be available. So here is the first one, which is getweather. You need to provide the location plus the date. Date is an optional parameter. And there's another function which is going to get us information related to flights. Now in this case you need to provide your origin and destination. You can have a number of different options. Here's the dummy data for that. For each of those we have the associated price and the both the departure as well as arrival time. Then there's another function for hotels in another function for get attractions. Now we're going to use another system prompt which is a bit more general. So we're telling it you Are a helpful travel planning assistant use the provided tools to help plan their trips, including checking weather, finding flights, hotels and suggesting attractions. Now we're telling it what the user is going to be looking for, but not providing specific function names the way we did it in the previous example. So again, the input is going to be user query. Now we will provide a list of tools here. So this is very similar to the structure that we have seen before. In each of the case we will need to provide what are the required input to the function that the model is supposed to generate. In this case, I modified the code. I am setting up a while loop because if there are multiple iterations that the LLM needs to do in order to generate response. So I want it to be able to do that. So again we will provide the model, name the messages, the tools that it's supposed to use, how many number of tokens that it can generate at max, right? And then we see or we check whether there were any tools that were used by the model or not. If it decide not to use any tools, that means that is the final response and we return that. Otherwise we provide the list of functions that are available and then check which function is being used by the model or agent that we are creating, get the corresponding data and then pass it on for the LLM to generate further responses. Okay, to really test it out, we have a relatively complex prompt. I want to plan a trip from New York to Paris for next week. So it's going to completely ignore the date. Can you help me with the weather, flights, hotels and attractions? So it has to decompose our initial user prompt and do parallel function calls before it generates the final response. Okay, so here are the responses that it generated. Based on the previous two calls, I can provide you with this summary of the trip. So it figured out what was the weather like? It's partly cloudy with 20 degrees Celsius. And we can confirm here that it actually is supposed to be partly cloudy with 20 degrees Celsius. Now it also gave us a list of different flights plus the hotels that are available, as well as the attractions that we have listed for Paris can go back to Langtrace. Now here you can see that it made the initial call to the LLM. There are three different calls in total for this specific loop. The second is the parallel function calls for each of the tools. So you can see that it's calling four different tools and getting the responses from each of them. And in the third step is combining everything together to generate a final response. So this was the initial input, that was the output generated. Then this is the input by using the parallel function calling. So here's the output that the model generated and that is the final output that we are seeing. In this case you can see that the total number of tokens are a lot more compared to what we saw for the previous simple example. So this can be extremely helpful if you are keeping track of how many tokens you are using. Okay, next, let's make it a bit more complex as well. Here it just gives us a list of let's say, flights that were available, hotels that are available, and we actually provided this list in the function definition or tool definition. But what if I ask it, I want to plan a trip from New York to London for the next week and then I'll fly to Tokyo and can you help me with the weather, flights, hotels, attractions? Just show me the best options for each category. So in this case we don't want this list, but it actually needs to figure out what are the best options available and just show us those. Now in here it says based on the results, here is summary of the best options for each category. So in London it's supposed to be 18 degrees Celsius and in Tokyo it's supposed to be rainy, but 28 degrees Celsius. Now in this case the we can fly from New York to London and from London to Tokyo. And if you look at the traces, so again, there were three function calls or three different calls to the LLM. The first one was parallel function calls to get those responses. The second one it get those responses and then provide all the information to the LLM to generate the final response. I further added a lot more functions in here with a lot more information to see whether it will get confused or not. So now we further expanded the list of functions that it's supposed to be using and again it was actually able to provide a very comprehensive list. So in this case I'm asking it to plan the trip that is going to include weather, flights, hotel, attraction, restaurants, transportation, currency exchange, and so on and so forth. And it holds pretty good with everything that I'm actually asking for. Now another thing which I wanted to try was the ability to do nested function calls. By nested function calls, I mean that it will need to make call to one function, then the output of that function is going to be the input to another function and so on and so forth. So in this case we're going to be using a movie recommendation system. So first we will need to provide the type of movie that we want to watch. So it can be action, comedy, drama, or sci fi. And here are a number of different movies that are available in this list. So it's going to randomly pick one of them based on what we want to watch. Then based on the movie, it's supposed to recommend a snack as well as where to watch it. So these movies are available on different platforms that we are putting it in here. Okay, so we're going to use the same structure, but in this case the system prompt is going to be you're a movie night play planning assistant. Use the provided tools to help user plan their movie night, including selecting a movie, recommending snacks, and suggesting streaming services. So the rest of the structure is exactly the same as before. Here are the three functions that it can use. The first prompt was can you help me plan a movie night with an action movie? So here's the responses. You're all set for the great movie night. You can watch Dark Knight on either Netflix or hbo. Let's see. So we asked for an action movie, it suggested Dark Knight. And if you look at Dark Knight, then it's only available on Netflix and hbo. And this is the snack recommendation that we have. And it also says don't forget to snack on some delicious Gotham City mix. Right. So it got that right as well. Now in the next one, I am asking for sci fi movie recommendation. So now in this case, I'm not asking it to help me plan or anything, but it's actually smart enough that it says you can watch Inception on HBO Max or Amazon Prime. Enjoy your movie night with a delicious dream layer tip. And here we can confirm that Inception is available on HBO Max and Amazon prime as well as that's the snack that it recommended. So pretty neat. It actually is able to do nested function calls, which is pretty impressive. Okay, so it was pretty good. But we used a relatively bigger model, which is the 70 billion model. But the question is going to be can the smaller model do all these things or not? And it's a little complicated. Okay. We're going to quickly walk through the same scenarios. So the first one was just use this function that we already told it to use. Right. So it's not going to be hard at all. And it's able to do that without any issues. Now, when it comes to parallel function calling, this is where it kind of struggles. So when we asked to plan a trip from New York to Paris, so it says, okay, I can help you with that. The weather will be sunny in Paris, which is Actually wrong. It was supposed to be cloudy. And then it says that there are a number of different options available, but it actually does not list those options. Neither does it list hotels, but it does list the things that you can visit or the tourist attractions in there. Now for the second prompt, it does provide a little more details, but I think it's still not as good as the 70 billion model. Next was to provide it with a lot of different functions and ask it for information related to those. Now here is where it struggles a lot. So it hallucinates a lot of information. So for example, it says that the forecast for Paris on October 23 is partly cloudy with the temperature of this. Now the temperature and the weather information is correct. However, we did not mention any dates and it just made this up. Again, for some reason it decided to use October 23, although the airline information is correct. Now it did hallucinate here in terms of events, there are events listed, but it says that there are no events listed between October 22 and 28. And it struggles a lot when it comes to nested function calls. Because when we asked it to help me plan a movie night for an action movie, it says it seems like the tools are not working correctly for action movie. Right. I tried suggesting a movie directly and it came up with Mad Max, which is an action movie. So I don't know like whether it actually used that function or not, but it's not able to use nested functions. In the second case it did suggest the Matrix, which is a science science fiction and it's available on PO Max or in Hulu. So seems like in this specific case it was able to use that, but again, it's very hit and miss. Okay, now let's talk about the function calling specific fine tuned models from Grok. This is based on the Llama 3. It's not based on Llama 3.1, but in my test this is the one that had the most trouble. So when I asked it what was the score of the warriors games? So it says to provide you with the score, I need to know the specific game you're referring to. Could you please provide me with the date or the opponent team? Right, so in this case we specifically told it that we're going to just tell you the name of an NBA team and it has to figure out the opponents from this function, but it's not able to do that. And it's kind of is a repeating theme when it comes to the other test cases as well. So for example, when you ask me to create a plan for a trip. It says sure, I can help you with that. Could you please provide me with more specific details of your trip? Right now it does the same thing when I added Tokyo to the list. So it says let's start with the weather. Could you please provide me with specific dates for your trip to London and Tokyo? Right? This could mean that I actually need to provide dates that this might be able to help me, but I just wanted to keep the test similar to what I have been doing for the other two models. Okay and when I asked it to help me out with a movie night, so it says do you have any specific action movie in mind or would you like a recommendation? Right so this specific fine tuned version of llama 3 for function calling is not doing great. Some final thoughts the Llama 3.170 billion model is definitely doing great when it comes to tool usage or function calling. It was able to do parallel function calls as well as the nested function calls which is probably one of the hardest one to do. 8 billion model I think it's not really ready for any serious function calling, so I won't even recommend using that usually for agentic use or for function calling. You definitely want a lot smarter models, so 70 billion is probably the best option. Now for some reason in my tests this specialized function calling model from CROC is not doing great at all. And actually the example that we are using here is based on an example that GROK provided in their notebooks. So not sure what exactly is going on here, but based on my test I won't even try to bother with this. And for observability purposes or tracing purposes, if you're looking for a platform that is open source at the moment, I recommend to check out Langtrace AI. If you found this helpful and want more technical content, make sure to subscribe to the channel. Thanks for watching and as always, see you in the next one.