GPT-4o mini - Can it be used for Agentic RAG?

GPT 4 Way Mini is the latest and the most cost effective model from OpenAI and it's one of the best performing models in its price range. But how good is this model for creating agents, specifically agentic rag? We are going to find that out in this video. I'm going to be doing a direct comparison with Claude 3.5 Sonnet, which is my go to model when it comes to agendic Rag. To make it more interesting, we will be using a practical data set and in this case that is going to be the Airbnb embeddings Data set from MongoDB. You will notice there are multiple different columns which has a lot of metadata. So this is an example of a practical data set that you can come across in the wild. And I'll also show you how you can set this up so that you can use it with an embedding model. Now, this data set already has text embeddings, but for the purpose of this video we are going to drop those embeddings. We're going to be using LLAMA Index to create our RAG pipeline. LLAMA Index has a very nice agentic RAG implementation, so we are going to be utilizing that for our LLM or agent. We will use both GPT4O mini as well as Claude 3.5 sonnet. And for our Vector DB we will be relying on Chroma DB. So to get started we will first need to set up different libraries. So we will install all the required packages. I'll walk you through them one by one. Next we need to set up our environment variables. So we will need anthropic API key. When we are using Cloud 3.5 Sonnet, you can download the data without needing hugging face token. But it's good to have that. So I'm going to use that as well. And both for the LLM as well as for embeddings, we're going to be using the OpenAI embeddings and LLM models. Next we're going to set up our LLM and embedding model for our LLM. In this specific notebook I'm using the GPT4OHM mini. I have another notebook which uses the anthropic 3.5 sonnet for OpenAI embeddings, we will be using the text embedding three small model. You can reduce the dimension of your embedding vector. By default, this has an embedding size of about 1500 dimensions. However, with this latest OpenAI embedding you can define your own embedding Size. This will both reduce your computation cost as well as the storage cost because you're storing smaller vectors. And the latest update also gives you the ability to do batch computation of embeddings. This also reduces your cost as well. So if you're doing offline processing, I highly recommend to look at batch embedding. And since we are using LLAMA index, so we're going to set both the embedding model as well as the LLM as default. Again, in another notebook, I am setting the anthropic model to be my default model. So here's that notebook where we are setting a different model as default model. Next, we're going to talk about data loading and processing. So, as mentioned before, I'm using the Airbnb data set from MongoDB. Now, instead of looking at the whole data set, we're going to just take the initial 2000 data points or entries. And the reason is that it's going to take a lot longer to process the whole data set, and that will incur a lot of charges as well. Then I'm converting this into a PANDAS data frame. This data set also has a text embedding as well as image embeddings as a part of the data. So we are going to drop the text embedding because we will be replacing this with our own embedding vector. So here we are dropping that text embedding column. So this is how the data set looks like. After dropping that one column, there are a total of 2000 rows with 42 different columns. And these columns represents different metadata associated with Airbnb listenings. So let's talk about how we do the embeddings for this data set. So first we take the data set, convert it into jsons, and then create a list out of all the different entries that are available. So we're going to have a document list with 2,000 entries. Then we're using some of those columns as metadata. So those contains amenities, images, host address, and a couple of others. These are going to become metadata. So essentially we're creating a dictionary with those specific values. Now, the main thing that we're going to embedding is the actual description that is provided by the user. But we also want to exclude some of these columns that are not useful at all. And here is the list of columns that we will simply discard from our data set. So in most of the cases, you will need to do a lot of pre processing on your data, and that's why you need to understand what columns or Entries are important for your specific application, and which are not. So we're going to drop a whole bunch of them. But for the most important ones, we are creating a metadata template. So this is basically the entries that we're going to be adding both to the embedding vector that we create that is going to become the metadata in the actual text as well. So if you look at it, this is the information that the LLM is going to see, and that has the name of the airbnb Listening. Then a summary. So this is basically the description. Any house rules, property type, room type, bedroom type, how many bedrooms are there, how many beds are there? Right. So these are the different information that is available through those columns that we put together into one text column or a blob of text that we're going to be using to embed. So we're going to compute embeddings for this, and then when we do retrieval, the LLM is going to see this text. Next, we need to split this into different chunks, and for that, we're using a chunk size of 5,000 characters, which is actually more than enough to take this whole blob of text and embedding embedded into a single embedding vector. So we'll use this sentence pleasure, and then embed each one of those documents or listing that we created. So with each split or chunk, we're going to take the text as well as the corresponding embedding embedded vector, along with the metadata that we create and put it in the node. And node is basically a chunk if you're coming from LangChain. Okay, so so far we only computed the embeddings. We haven't stored them in a vector store. So for vector store, we are using the Chroma DB vector store. It's going to store everything in a directory called Chroma db. The collection name is going to be airbnb Listening. And we're going to use this as a default vector store. In order to use that, we'll need to bring it to our storage context. Storage context is used by llamaindex to define the default values of both the vector stores and other corresponding hyper parameters. So we create our storage vector, we bring in our nodes. This will create a vector store which is going to contain the actual text chunks, which is basically the listening in our data set, along with the embedding vector that was computed in the last step, plus the metadata that we created out of the listening. Okay, so so far we only created our vector store or index. The next step is to start creating our agent. Now, the way Llama Index implements it is that it creates a set of different tools that are going to be available to our agent or LLM. And for that we're going to be using the query engine tool class. So you provide your query engine, which is essentially the vector store that we created, and it's going to return the top five similar chunks or similar listening in this case. And we say that this is going to be a tool that is going to be available to our agent. We call this knowledge base and the description is, provides information about Airbnb listening and reviews use a detailed plan, plain text question as input to the tool. Right now you can have multiple tools if you want, but for this, for the sake of this video, we're going to just use this single tool. And in this case I am embedding everything into a single vector store. But you can have multiple different vector stores that can contain documents from multiple different fields or specialization. And then the agent can determine based on the query which index to use. I have a couple of videos on that. I'm going to put those in the video description. Okay. Once we define the tools, the next step is to create the actual agent worker. Llama Index implements function calling agent worker for that. So you provide the list of tools, which is query engine tool in our case plus the LLM. The LLM is GPT4 on mini and we are going to enable verbose so we will have access to the internal thought process. So we create our agent worker and then set this as an agent. Now, for the user interaction, you just need to call the chat function on it if you want to give the agent memory as well. So here is a one single prompt, tell me the best listening for a place in New York and GPT4. Oh, Mini is not great as an agent and you can see it from how it's processing it. It says added user message to memory. Tell me the best listening for a place in New York. This is the actual user input. Now the agent is supposed to rewrite the prompt so the way it wrote the prompt was best Airbnb listening in New York. And then it used the tool to try to figure out what is the best listening. So it says the best Airbnb listening in New York is charming bedroom in East Village. And it also returns the actual description that is provided in the listening. But the thought process in itself is not great in comparison. When I ran the same Query through Cloud 3.5 Sonnet as an agent, here's what we get. So the LLM responses certainly to provide you with the best listening for a place in New York, I will need to use the Knowledge Base tool to gather information about Airbnb listenings in New York. So it says let me do that first and then it does the function calling, but here is the input it's providing to the agent. So basically it rewrote the prompt for us and the prompt is what is the best Airbnb listening in New York City? Please provide details about its location, amenities, price and guest reviews. So now here's the output that it got and based on the output it generated a much detailed response which is pretty great compared to what we are seeing from GPT4 or Mini. Now I asked it what is the worst one, right? So here's the response. So it again created a very simplistic prompt based on the user input and the response is also again, not that great either. Here is a response from 3.5 Sonnet for a very similar prompt. Now the Miami is not actually present in the listening, so it says. I apologize for the confusion. It seems that our knowledge base does not have specific information by about Airbnb listening in Miami, but it says however I can provide some general insights about how vacation rentals might differ between the two cities. So it says location and environment, type of accommodation, amenities, price, seasonality and some other characteristics. So it's a very detailed and I think a highly accurate Response from Cloud 3.5 Sonnet. So in general GPT4 or Mini is a pretty great model, but I think it's not great for agentic workflow. You still definitely want to use a much more powerful and bigger model when it comes to agent orchestration. I'm going to put a link to this collab in the video description. Let me know what you think and how does it perform in your own agentic workflows. I hope you find this video useful. Thanks for watching and as always, see you in the next one.