How Good is LLAMA-3 for RAG, Routing, and Function Calling

In the last video we looked at PHYT3 beyond benchmarks. So I thought we'll repeat the same thing for llama3 as well. So in this case we're going to be testing Llama 3 on rag, its ability to do query routing and tool usage or function calling. Now function calling is not officially supported in LLAMA 3, so we're going to be using the GROQ implementation. For that to test racing, we need a source of data. So for that we're going to be using this article from Verge. The synthetic social networking is coming. This is the same article we tested with Phi3. As always, link to this notebook is going to be in the video description. So first we need to install all the required packages in order to run this notebook. We're going to be using llamaindex to implement the RAC pipeline. Now first we need to load the contents of this web page. So for that we are using the Beautiful Soup web reader. It's an amazing library for reading HTML content. We provide the URL of the website that we are working with and it will convert it into a document object. Next we need to load our LLM. So we're going to begin testing it with Llama 3.8 billion version. But later on I'll show you the results for 70 billion version as well. And in order to load the model we are going to be using the amazing GROK API which is one of the fastest or actually the fastest inference API currently available on the market. So here I have my all API keys as secrets, so I will just enable access to the GROK API key. Right and we are good to go. So it will load my GROQ API key and we create this GROK LLM object. The model name right now we are providing is llama 38 billion. And later in the video I'll show you how to load the 70 billion as well. You just need to change the name of the model. Alright, if you're doing rag, we also need embedding model. So for this we are using the open source embedding model which is the BGE small English embedding. If you want to use Mistral embedding, OpenAI embedding, even Gena embedding or Cohere has its own embedding, so you can just provide those here. Okay. And we're setting both the LLM as well as the embedding model in the settings of Llama index. Alright, next we need to create our vector store. So for this video we will be creating two different vector Stores and the reason will become apparent when we are looking at the query routing. So first we create a vector store which will convert a document into chunks, compute embeddings for each chunk, and then store those in this vector store. The other vector store or actually summary index is basically summarized version of the document. So these are two different options that we have and we are going to switch between them when we are doing query routing. Okay, next we have some helpful logging code. This will keep track of everything that is happening. Now let's look at how the query works. So we create our vector store and based on the vector store we will create a query engine and then we pass on our query to this query engine to get responses from the model. So the first question is how do OpenAI and Meta differ AI tools? And here's the response that we get based on the query that is going to be passed through the vector index that we just created. So the response is OpenAI tends to present its products as productivity tools, simple utilities for getting things done. Meta on the other hand, is in the entertainment business. And the answer is pretty accurate based on the context that we have provided. Now one thing I have noticed is that the Llama 3 models actually tends to generate pretty short and concise responses compared to what we saw for Phi 3 models in our previous video. Alright, so here's another one. What are the new features added by OpenAI to ChatGPT? So they are talking about adding voice and image generation or image upload ability. So it actually correctly identified that it has added two new features to to ChatGPT. The ability to interact with its large language model via voice and the ability to upload images and ask questions about them. In the previous video, 5, 3 was actually confusing some features that were added to Meta AI with features added to ChatGPT. But the 8 billion version of Llama 3 does a pretty amazing job at retrieving the correct information. Now I asked the same question but regarding Meta AI and actually correctly identifies that Meta AI unveiled 28 personality driven chatbots to be used in Meta's messaging apps. This is actually correct. So this is pretty good. Okay, next we're going to look at its ability to do query routing. So what exactly is query routing? Let's say you have multiple databases. For example, you are a teacher and you create a vector based for mathematics. The other is for physics. And let's say the other one is for chemistry and the user is trying to learn something so that they ask a question. Then this query router has to decide which Vector store to use in order to generate an answer. So if the question is related to physics, it will retrieve information from the physics vector store. If it's related to mathematics, it will retrieve data from there. Right. And you can expand this to other use cases as well. For example, if you're working in a company, there are different, let's say, departments. You can create vector stores for each department and then depending on the query, as well as the user authorization level, you can detect or you can select which vector store to use. Right. So it's a very important ability to have. In this case, the query router is basically an LLM which will decide which vector index to use that is going to be selected and is going to be used to generate responses. Now, to keep it simple, so we create two different tools and we're going to be passing on to our query engine. The first one is the vector store that we created and it's useful for searching for specific facts. The second one is the summary tool or the summary index that we created, which is useful for summarizing an entire document. Right. So these are going to be the two different vector stores or vector indices. Right. And the query engine tool using the LLM needs to determine which one to pick, depending on the question that user is asking. Okay, so we created a multiple selector. So we provided a list of tools to the router query engine that we are going to create. And depending on the context, this has to select either one or multiple of these available tools. So the tools in this case is either the vector index that we created or the summary index that we created. Right. And can select multiple if need. It needs. So the first question was what was mentioned about meta? Summarize what is mentioned about other companies in the document. Now it actually selected only the first tool to use, which is the vector index. And the reason is the question is asking to summarize what is mentioned about meta, which implies searching for specific facts, making choice one the most relevant. Right. So it thinks that since it's talking about meta, so it has to figure out very specific information. Right. And here is the response that it generates, not the best of responses. It seems like it it kind of copied a sentence from the provided context. Right. But at least it was able to correctly identify that we need specific information about meta. Now, for some reason it completely ignored the second part of the document. So which is a bit concerning. Then I kind of expanded this. I said summarize what was mentioned about OpenAI and in this case it selected the second option. So it's going to be using the summary index. And the reason is the question is asking to summarize what was mentioned about OpenAI which implies, which implies summarizing an entire document, making option two the most relevant choice. Right. And then it provides a summary of what was mentioned about OpenAI and the summary is actually pretty good based on the provided context that we have. Okay then I actually wanted to see how a 70 billion model is going to respond to. So here I changed the model to the 70 billion model. We are still using the Gronk API. Now in this case the responses for same queries are actually much better. So the first part was what was mentioned about meta and rather than simply copying a sentence from the document, it actually kind of summarizes everything and talks about the 28 percentage determined chatbots that are created by meta which are going to be used in its messaging app. And then it also says as for other companies, OpenAI is mentioned as announcing updates to ChatGPT including voice features that allows for user to interact with language model via voice. This feature gives ChatGPT a more human like personality. Right, so this model is definitely doing a much better job at retrieving information when there are multiple parts the query. And it does a pretty good job again at this summarize what was mentioned about OpenAI as well. And in this case it is trying to use the summary index rather than the vector store. So as you would expect, the 70 billion model does a really good job at query routing for more complex queries. Alright, so the last part we want to look at is its ability to do function calling. Now llama3 doesn't officially support it, but CROC has an implementation for tool usage which is basically function calling. Okay, let's first try to understand what function calling is before we look at an example. So there are scenarios in which you ask the LLM and it doesn't has the ability to provide you an answer. So in that case using some external tools is going to be helpful. So for example, if you ask a complex mathematical question, it's better for the LLM to use a calculator rather than doing the mathematics by itself. Right. So how does it work? So first you ask a query, the LLM has to decide whether it needs to use an external tool or not. If it thinks it doesn't need to use an external tool, the LLM will generate a response for you. Right. So there are still two calls to the LLM. Now in case if it needs to use an external tool, then first it needs to find what the appropriate tool is for that specific task that's basically picking the tool. Then it needs to make a call to that function. Usually the tool usage is implemented as a function. It will get a response from that function and then that is fed into the LLM and the LLM is going to generate a response. Now Groq has implemented the same thing as a external loop for llama 3 and the mixtral Moe models. So in this case we're going to be using a notebook provided by Groq. So you first need to install the Groq client for the Python. We set the API keys exactly the same way we did it before. Now I'm using the smaller model and it also does a pretty good job. The bigger model is definitely much better. The dummy function that we are going to be calling is we want to get scores of NBA games. So there are different scenarios in which we have different teams and there are different associated scores in this, right? So depending on the input, it will pick a team and give us the final score. Alright, so here's how the main loop looks like. So this is the first system message that goes into the first call of the LLM and the system messages. You are a function calling LLM that uses data extracted from getgamescore function to answer questions around NBA game scores. Include the team, their opponents in your response. Okay, so the user asks a question that is going to become part of this initial message or initial prompt, right? Here we are defining the tools. So you can have multiple tools, but we are just sticking to the one that they have provided in their example. And the main thing here is to make sure that you include very detailed description of what different tools or function do. Right? Because based on this it will select which tool to use, very similar to which index to use when we were looking at the query routing example. Okay, and then what is going to be the output? So that is the team name that the model is supposed to return. Okay. And then like some description around what the tool does. So for example, it's supposed to use the team name, that is going to be a string and the name of the NBA team. Okay. Now so this is going to become the system message or the initial query that will feed to the LLM. So here we are making the first LLM call right? Now in this case, the way the GROK has implemented is in the response. You can check whether it decides to use any tool or not. If the tool usage flag is true, that means that it wants to use a tool. So you'll have to Make a second call to the LLM. Right. So in this case we first check if the tool usage is true or not. If it's true, then we look for what tool was used. And right now we have only one option. So it will just make a call to that tool. It will get the response from that function that is fed again as an input to the LLM itself. So that will become the second query. So basically we are talking about this step. Get a response fed into the LLM to get a final response and that second response is the one that is going to be shown to the user. So for example, here's the first query. What was the score of the Warrior game? So it goes through a step by step process of whether it needs to make a function call or not. So it actually decides that I need to make a function call and the team name that it selects based on what is provided in the prompt is Golden State Warriors. Right. And it will get a final response. It says according to the call, the Golden State warriors and their opponent is the LA Lakers. The final score was this in the in the favor of Warriors. Right. So it actually got the response correct. Now if I ask something irrelevant to the functions or tool usage, so it actually goes through the whole thing again. But in this case it says, I'm glad you're interested in knowing the purpose of life, but I would like to take a different approach. How about talk about the game between LA Lakers and the Golden State Warriors? Right. So since it's a smaller model, it wasn't able to actually correctly deduce that there is absolutely nothing about the NBA games and it decided to use the LA Lakers as an input to the function call. But if we replace it with the bigger 70 billion model then for the same query, now it doesn't have any problem. So it says it tries to figure out whether I need to use a function call or not. It decides that I don't need to use a function call because it's not really relevant to the function that is available or tools that are available. So it says as a final response. That's a deep and complex question. I'm happy to provide some insight. And it gives us pretty detailed answer. And it doesn't do any tool call, which is pretty amazing. So the bigger model is definitely a lot smarter, which is expected. But I think even the smaller 8 billion model is pretty capable. But there might be some situation in which you definitely want to try the bigger sibling. Okay, so this was a more of a practical use cases of these model and its abilities. And as I said, both these, these 8 billion as well as the 70 billion models are pretty amazing. Meta has done a really amazing job with these models, so I would definitely recommend everybody to check out Groq. They also provide a free API which is extremely fast. I hope you found this video useful. Let me know if you want me to do comparison of other models for similar tasks. I hope you found this video useful. Thanks for watching and as always, see you in the next one. Thanks for watching.