How Good is Phi-3-Mini for RAG, Routing, Agents

The PY3 small language models are pretty amazing when it comes to benchmarks. However, are they actually any good in practical use cases? That's what we're going to be trying to figure out in this video. So I personally work a lot with Rag, so I put together a list of different applications that we want to test this model on that will include simple RAG, query rerouting, decomposition of complex queries, and its ability to work in an agentic framework. Now, to test all this we need data, and for that we're going to be using this article from Verge. The synthetic social network is coming. It talks about different features that were introduced by OpenAI and Meta AI. So let's look at the code and see how good this model is. This notebook was derived or adopted from a notebook provided by Llama Index team. I'll put a link to that. So first we are installing all the required packages. We will start with drag and then we'll go down the list one by one. So first we need to download the data. For that we're going to be using the Beautiful Soup package, which is a python library for reading HTML files. We provide the URL of the article that I was just talking about. This will download the document. We can look at the document content. So it has the URL followed by the actual text from the document. So we can see that Beautiful Soup was able to correctly load the URL. So this is good. Okay, since we are talking about Rag, so we need to load the LLM and as I said, we're going to be testing the PI 3 mini 4K instruct version. So this function that you see here, it will convert the message into the proper prompt template that you have. You can have multiple messages and it will put together the system role, user role and assistant role in proper places. We're using the Hugging Face LLM function within Llama Index to load the LLM from Hugging Face. So you provide the repo id and then since we're downloading this from Hugging Face, this model does require to enable Trust remote code. So you'll have to do that. And we will be using the same tokenizer. Now you can also provide the query wrapper prompt. So basically this is a prompt template that is going to be used by the model. Now, one thing which I highly recommend is to use a higher quantized version of the model if possible. So in this case I'm using the 16 bit precision. You can use the 8 bit precision as well. I won't recommend using 4 bit because it's such A smaller model that will highly degrade its performance. Now for Rag, we need to create a vector embedding. So for this we are using the BGE small vector embedding. These are English specific embeddings and we are setting both the LLM as well as the embeddings in the settings of Llama Index. If you're not familiar familiar with Llama Index, I have a series of videos. Links are going to be in the video description. Now we're going to be creating two different vector stores and this will be useful when we are doing our query routing. The first vector store is based on the chunks of the documents. So we're using the vector store index. It has default values for chunk size, chunk overlap and how it does the splitting process. So I'm not changing any default values in here, but basically it will take the document plus the embeddings and it will create a vector store for us. The second index that we are creating is summary index which is used for summarization of the documents. So we will create two separate embeddings and both are created using Llama Index. Now this section just sets up some utility function for locking. Next we're going to do simple retrieval. On top of our vector index that we created we're using the compact response mode. So this is very similar to stuffing of context in LangChain. So it will basically take all the relevant chunks, put them together and it will do a semantic search based on the query that you provide. So the query is how do you OpenAI and Meta differ on AI tools? So this will just do stuffing of the context and generate a response like very simple RAG implementation. And the response is OpenAI and Meta differ in their approaches and focus on AI tools. OpenAI tends to present products as productivity tools, right? Then on the other hand, Meta, which is in the entertainment business, is building LLMs and has developed personality driven chatbots for its messaging application. This is a pretty accurate response based on the contents of the documents. Okay, what happens if you ask a little more complex queries. So the query is what are the new features added by OpenAI to ChatGPT? Now this is where things get interesting. So it says wise feature. ChatGPT now allows user to interact with the large language model via voice. This is accurate. But then it says personality driven chatbots. So OpenAI has an unveiled 28 personality driven chatbots powered by LLMs to be used used in Meta's messaging apps. So you can see it's actually confusing. Some of the features introduced by Meta AI and it says that those are introduced by OpenAI. But then in the next kind of part of the sentence says that they're being used by meta AI. Right. So this is some hallucination that is happening now. It talks about image generation, the article talks about image generation tools specific to meta AI. So it's again confusing this feature that was introduced by Meta AI with OpenAI. I think if you use re ranking techniques that will probably help the model. But in this case we are just looking at Rack pipeline, Right? If you want to learn about more advanced techniques, I'm putting together a course linkage is going to be in the video description. Make sure to sign up for that. Now we can ask the same question, but instead of putting all the chunks together, we can use this tree summarize mode. So basically it takes the first chunk and recursively creates summary of each chunk and put them together to get a response. Right. So if you do that, the responses much better. But the last part, again it confuses the Meta AI feature with ChatGPT feature. Okay, so it seems like it does a pretty decent job when it comes to simple queries. But when it's more complex queries which has conflicting, I think, information in different chunks, then the model is not able to figure out and it might have some limitations. Right. Okay, now let's look at its ability to do query rerouting, which is actually another application that is getting a lot of traction. But first let me explain the concept of query routing or rerouting. So let's assume you have multiple vector stores and each vector store has different sort of information. Now you will provide a detailed description of each vector store to this query router that will take the user query and based on the description of each vector store, it will decide which vector store to use. And then based on that vector store, the LLM will generate a response. So in this case, the model needs to be able to look at the description of a vector store and determine which vector store to use for information or retrieval. Right. So llamaindex implements this query engine tool. You can use tool metadata. So what we are doing here is we are creating multiple tools. So the first one we are calling vector tool, that is our vector index. And in the description we need to provide name plus the description. And this is the most important piece of information. You want to make sure that the description is highly detailed so that the model can actually choose the appropriate vector store to use when it comes to is doing query routing. So the first description is useful for searching for Specific facts. And the second one is the summary index that we created and the description is summary or useful for summarizing an entire document. Right. Now let's say we ask a question, but in this case we are specifically forcing the routing query engine to actually select one of the tools to use or one of the vector store in this case. Right. So for that we are using the router query engine. It's a class provided by llamaindex and we are going to provide a list of vector stores. Then here we are setting that you can select only one of the vector store. The query is what was mentioned about meta. So in this case it decides to use the first index, which is the vector tool and it says useful for searching for specific facts. Then it's able to actually provide a pretty decent answer. So it says meta, which is the entertainment, which is in the entertainment business, is building LLMs and has found its own uses for generative AI and voice. Right? And then it talks about those 28 personality driven chatbots. So this is so pretty good so far. So I wanted to make sure that it's able to actually retrieve specific information. So I said how many personality driven chatbots were introduced by Meta? So very specific piece of information. And again it uses this vector index that we created and it says 28 personality driven chatbots were introduced by Meta. So pretty great. Now what happens if we do multiple selectors? So in this case, basically we are enabling it to select multiple tools if needed. And now the model has to decide whether to combine multiple tools or just use a single tool. Right. So it becomes a little more complicated. And now the question is what were the main features introduced by OpenAI? Summarize the features introduced by other companies mentioned in the passage. Right. So we're not talking about meta only. We are letting the model decide which other companies were mentioned in the passage. Now in this case it decides that it's going to be summarizing an entire document. Right? Because that's the most relevant to the task summarizing the main features introduced by other companies mentioned. Right. So it decided to use the summarization tool and the response that you get is here. So OpenAI introduced a voice feature for chat GPT which is actually accurate, allowing user to interact with the AI via voice. Okay, so the information so far is accurate. Now it also kind of hallucinate here or miscategorize the information. So it says open a also in wheel 28 personality driven chatbots for Meta, Meta's messaging app. Okay. So now the main thing is that, okay, even though it got it wrong, but the information is actually there, right? Although it's related to Meta, when it comes to Meta, it says introduce 28 personality driven chatbots for its messaging apps. Okay, so it got this right, but somehow it also thought like OpenAI introduced that. However, the biggest issue is here. So other companies mentioned in the passage. So it talks about Tesla announced a new model, the Tesla Cybertruck, which is a futuristic electric truck with a unique design. Now this information is correct. However, there is absolutely no mention of Tesla in the document. Similarly, there is absolutely no mention of Apple in the document. So it simply hallucinated this information and included it, which is pretty bad. Okay, so based on this, can you use it for query routing? The answer is potentially yes, but for complex rag, probably no. Now another relevant task is sub question query engine or basically query decomposition. So if you have a complex query, something like this, what was mentioned about meta, how does it differs from how OpenAI is talked about? Right, so so now the query decomposition works like this. You will divide this query into multiple queries and then extract information based on each query individually and then combine those together. Right? So this model is actually able to do a pretty good job when it comes to decomposition of complex queries. So for that we're going to be using the subquestion query engine. Again we're going to provide both our indices as tools here. Like think about this as function calling, right? But it has to automatically determine which tool to use. The query that we provided is here. Now based on this it says generating three sub questions. So it decided to actually decompose this query into three questions and the question it comes up were pretty decent. So it says the first question is what are the key points mentioned about meta in the documents? And then it tries to retrieve all the information that is mentioned regarding meta. Okay, Then the second question that it came up with is this, what are the key points mentioned about OpenAI in the documents? Right, so this is actually a query that you would expect it to be able to generate based on the original query. And then using that query it does use the vector search. So it's actually using the vector index to retrieve that information. So pretty good. Although like it got this personality driven chatbots wrong again. Now the third question it comes up with is how does the information about meta differ from information about OpenAI? So now basically it talks about this part and it will look at the information that it retrieved from Meta plus OpenAI and then it will see like how they differ from each other. Right, and here's the final response. It says Meta has a venture has ventured into entertainment industry by creating its own generative AI and voice technologies. Right? And there are 28 personality driven chatbots. Okay, so it provides information about Meta, then talks about OpenAI. So OpenAI ChatGPT strives to be more human, like conversational AI. Both companies are exploding AI's role in social networking, but their approaches and applications vary. So again, it does a pretty good job at query decomposition, but its rag abilities for complex queries are limited. Alright, next let's talk about whether you can use this to orchestrate agents or not. Right, so for creating an agentic framework, we're going to be using the React agent React agent from the core agents within LLAMA index and we are going to pass on both our vector stores as tools in this case. Okay, let's first ask a simple question which is not related to any of the tools. So the question is hello. Then it goes through the internal thought process. It says the user is greeting, so no need, no tool is needed to respond. And the answer that it generates is hello, how can I assist you today? Right, so basically based on looking at the description of these tools, it decides I don't need any of these. So I'm going to just generate a response. Now if you ask a question, what is mentioned about Meta? How does it differ from how OpenAI is talked about? Right now it has to actually look at these tools. So you see and it says I need to use a tool to help me answer the question. So it does a vector search and the query that it comes up with is Meta versus OpenAI is comparison. Right? And then based on that it actually goes through the whole process of thinking. Right, and here's the final response. So Meta is primarily known as a social media company that focuses on building Metaverse while OpenAI as a AI research lab known for its work on AI safety, ethics and advanced AI models like GPT 3.5. Now the article talks about ChatGPT, but in this response it actually doesn't really talk about that at all. Okay, but now what about a more complex question? What are, what were the new features introduced by OpenAI and who introduced the personality driven chatbots? Right, so again it goes through its internal thought process. It is going to use vector search and here's the final response that it generates. Matter introduced new features in the form of personality driven chatbots to be used in the messaging apps. So okay, this is the second part of the query and actually got that information. Right? But then when it comes to OpenAI, it doesn't really talk about the new feature at all. It Simply says what OpenAI does, right? So again, for like more complex tasks, it does seem to have very limited abilities. All right, what if we were to use some other tools other than vector stores? So here is a quick test. In this case we're defining a couple of functions. One is multiply, the second one is addition. Now in this case the doc string is going to be used as description, right? So this is a specifical set of tools which you can define as a set of functions. We're going to create a react agent. Again we'll provide both our tools in here and the LLM is going to be driving or orchestrating these tools. Now when asked what is 121 plus 2 multiplied by 5? So it basically says I can answer this question without using any tool. And since it's a simple mathematical operation, it actually tries to solve it, but it comes up with incorrect answer. Now interestingly enough, when it even solves it, like it goes through a whole thought process of kind of unrelated things, right, which are not accurate. Now one thing I realized is that whenever I tried to use mathematical operations as tools. So for example, here's another one in which I said what is 20 into 2 plus 5 actually does a computation by itself and comes up with the correct answer. So when it comes to mathematics, it feels like the model doesn't want to use any tools. But in one of my previous videos I actually used the Mixtral 822B and that model decides to use tools whenever it has to do mathematical operations because I think those gives you a lot better responses and it has a lot better thinking ability. Okay, so these are my quick tests. Now what are my final thoughts? So it, it's definitely a capable model when it comes to simple rack, but it seems like it does hallucinate a lot. So for complex questions probably it's not a good option. It does seem to be able to do query routing. So for definitely you can use it for that, but I think the underlying rag mechanism is going to be hard for you to figure out and I would say it's probably not ready to perform agent orchestration for complex tasks because I think at least in my test it has very limited abilities. Okay, so this was a quick test of different abilities. So I'm going to be doing a lot of these tests with the new LLMs, apart from doing testing on a specific prompts that I use. So let me know if you like this approach. I hope you found this video useful. Thanks for watching, and as always, see you in the next one.