Multi-Modal RAG: Chat with Text and Images in Documents

We are going to continue our discussion on how to build multimodal RAG systems. In the previous video we looked at different architectures that are possible for building a multi modal RAG application which combine both images as well as text data. In this video I'm going to show you how to build an end to end system using GPT4O and Llama Index. This is going to be an extension of our previous video. Now here is how the architecture is going to look like for this specific case. So first we're going to collect data. Again, this is going to be a combination of both text and images. For text, we're going to chunk those, create a separate text vector store. We'll take each image separately, run them through an embedding model, which is going to be a clip model again in this case, and create an image vector store during user query time. We're going to process the query and compute embeddings weight and then we'll do retrieval based on the text vector store as well as on the image vector store. We will combine them together to augment the context for our LLM, which is GPT4O in this case, and that will generate responses which are going to be returned to the user. So here's a quick overview of how this video is going to look like. So first we'll simply talk about multimodal RAC system. Then we'll talk about the setup, installation, how to prepare the environment. In this case I'm using Google Colab Notebook, but you can run this on your local machine as well. Then we'll quickly talk about data collection and preparation and then how to create multimodal indices based on both images as well as text. Then we will implement a multimodal retrieval pipeline which is going to be retrieving both images as well as text chunks based on the user query. And we're going to wrap everything around in a single pipeline that was going to use GPT4O to generate responses. So in the multimodal RAG system you combine both text and image data to enhance the capabilities of the large language models. So there are four different steps, very similar to the traditional RAG systems. So the first one is indexing, where we combine both image and text data and store them in separate vector stores. I'll also show you how you can use the multimodal model like GPT4O to generate description of images that can potentially be combined with the text chunks to augment your vector store. Then you do retrieval. So basically the user provides a query and you retrieve both the Text and images that are relevant to your query. And after that we do augmentation. So the retrieve information is used to augment the input to the LLM and the LLM will generate a response based on the augmented input plus the user query. Okay, so let's talk about the setup. We will be again installing the Clip model. This model is going to be used to generate the text, the image embeddings. In our case, we will use OpenAI for our LLM. Quadrant is going to be the vector store because we are looking at a multimodal vector store. And Quadrant is one of the vector stores that supports both image as well as text data. If you are looking for another alternative, ChromaDB is another open source vector store that supports both, so that can be an option. And we will also install some other packages that are going to be just auxiliary packages. Okay, so first we need to import the packages that we will need. We need to set up our OpenAI API key. I am again using Google Colab Notebook Secrets. Actually person who implemented this needs an award because I think this makes life so easy. Now after that we are creating a couple of different folders. So one is input images. This is a folder that is going to contain some of the input images that I'm going to provide. As an example we're going to see like how you can use GPT4 or to generate descriptions. Then another folder that we are creating is called Mixed Wiki. This will contain both images as well as text that we are going to be reading from a Wikipedia page. So first we check if those folders do not exist. This script is going to generate or create those folders for you. Now to show you how you can use GPT4O to generate description of images, we are going to download a number of different images. So this, these are images related to different Tesla vehicles. Actually these are, I think all of them are related to Model Y. So for example, there are different configurations of Model Y and these images have the specs for different configurations. So this one is performance. I think the first one was long range. And you can see there are information related to the weight class, what is the speed and I think also the pricing is here on this, this specific image. Now this small script will just display all the images contained in a folder. So these are the images that we looked at. Now the first example that I'm going to show you is how to use GPT4. Okay. To generate description for each of these images. And you can use these descriptions as text chunks and put them in your vector store. We're not going to do that that in this specific video, but in a subsequent video I'm going to show you how you can perform that process. So here we're providing the input images folder, which contains all the images that we downloaded. And then we're using the OpenAI multimodal function from Llama Index. So they have written a function that simply wraps GPT4 or the OpenAI API for multimodal applications. So the first prompt is simply generate detailed text description for each image. So basically it takes one image at a time by reading those images from the directory and feed it into the LLM to generate responses for those. So here is a detailed response to image number one. I think this is one related to the specs. So basically seems like it did OCR on the image and generated a text response. The second one is structural diagram of the vehicle. This was the think this specific diagram. So based on the diagram, it generated a description of what it sees in the image. Right. So normally if you're using, if you're running these images through a multimodal model like GPT4, okay, you can use these text description as text chunks to augment your existing vector store. But for the purpose of this specific video, we're going to be sticking to using clip to generate embeddings and that's going to be useful retrieval purposes. Then we have this helpful function that is going to download images from Wikipedia pages. So in this case I am providing links to different articles on Wikipedia. So for example, this Tesla model xy, then Kia Rivian. So these are different electric vehicle. We want to get the information regarding all of those. We run that script and we will download images as well as text from those Wikipedia articles. Now in my case I was running into this too many requests. So you could potentially add a small pause or that will probably help you not getting into this specific error. But I was able to download some of the images so we can look at some of them. For example, this, this seems to be, I think Model S and it downloaded data for Rivian Model S X. And we're going to also download a 10k form for Tesla. So here a truck that Rivian is creating for Amazon. So there are both Rivian images as well as Tesla images. We have the images. We're going to create two different vector stores and then combine them together using a multimodal vector store. So the first one is going to be for text and the second one is going to be for image. We're going to Be using Quadrant as I said. So we create the Quadrant client. Then if you're trying to create a multimodal vector store using Quadrant, you just need to provide the names of two collections. So we create one collection for text. The second collection is for images. So think about these are two different tables in your database. One belongs to the text chunks and the other belongs to your images. Since we're using llamaindex, so we will need to create storage context. If you're not familiar with this idea, I have a couple of videos on Llama Index which really explains the concepts. So I'm going to put links to those. Storage context basically contains all the information regarding where to store your vector store, what type of embeddings to use, what are going to be the chunk size, and so on and so forth. So we load the data from this mixed wiki directory that contains both text as well as the image data, and then we create a multimodal vector store based on all the documents that are contained. The storage context is going to be the one that we created using both the image as well as the text collection. So this basically combines embeddings of both the text as well as your images. Next we will create a rank pipeline. Now in the first part we're going to just look at what kind of retrieved context we are seeing. And then in the next part we're going to feed it into the LLM to generate our final responses. So we need to set up some hyper parameter for our index for retrieval. So we take that index, create a retrieval on top of it, and we say that for every query it's supposed to return the top three text chunks and the top three images that it can find, which are similar to the query that the user is providing. So we write a helpful function which basically takes that retriever gets the user query and we're just inputting only the top 50 tokens. You don't really need to put this, but this was just to show you for extremely long user queries. You can put how many tokens you want to process. This will just help you save some money. Right, so we get the retrieve results. Now the retrieval results are going to have two different parts. One is going to be text chunks and the second one is going to have images. So we take or separate both of them. So here we are looking at the images and putting them into retrieved images list. And the second part is the text that we are going to be displaying, right? And if there are images, then we just show those images, right? So Here is a simple prompt. What is the best electric sedan? And you can see that these are three chunks that are retrieved by the text embeddings. So the first one is the Tesla Model S is battery electric executive car with a lift back body style built by Tesla. The second one talks about Model X which is not a sedan. But if you look at the images, it only retrieved images of Model S, which is pretty good because we are specifically talking about a sedan. So in the last step we can just take all this contacts, put it, put it together and, and feed it into GPT4 to generate a final response. So here is how it looks like we will be using prompt template. And in that prompt template we say that okay, context information is below. So this is the context that the model is going to get. And then given the context information and not prior knowledge, answer the query. So we, we, we ask it to answer the specific query that the user is providing. Now in this case we also want to run both the images through this whole process. So we're going to create a prompt template. Then we are going to run that prompt template on the index that we created. In this case we're going to be using GPT4O as our multimodal model. So again, keep in mind that this is not the normal GPT4O but the multimodal version that Llama Index implements. So it's going to receive both the images as well as the text to generate the final response. So this pipeline now receives the user query and the LLM along with the context that is going to be retrieved by our index. When we run our query on the query engine that we just created, we're going to get a response and then we also look at what images were retrieved as part of the context. So we're going to display those as well. So here I am asking it to compare the design features of Tesla Model s and Rivian R1. So the responses that you are going to get are going to be based on the text as well as the images that are retrieved. So here it talks about Tesla Model S. So the body style, powertrain and design evolution, range, interiors, additional information. And it looks at the exact same features for Rivian R1 as well. And I think at the end it give us a small comparison or summary of whatever it has found. So it says that Model S is a sedan which is focused on performance, range and luxury, whereas the R1 or R1s are designed as a pickup truck and SUV respectively with a focus on versatility, off road capabilities and utility. Now we can also look at the sources. So basically the text chunks that were retrieved as well as the images. In this case it only retrieved two images. One is that of Rivian R1 and the second one is Model X and the second one is Model S. Now for this specific project, we combined both images as well as text chunks and created an end to end RAG system or RAG pipeline. Now you can further improve this by using agentic rag. That's going to be a topic of our subsequent video. So we're going to look at how to create agents that can perform RAG for you. That's extremely helpful in situations in which you cannot find the answer in a single shot. So for example, in the first case if it's not able to retrieve the proper chunks, then the agents is going to do a second pass of retrieval by changing the prompt. So that will change the retrieved context and it can also look at how good the responses or the chunks are before generating final responses. So I'm going to be creating some videos around how to do agentic rag. I think that's going to be a topic which you will find extremely helpful. Now there are a number of different approaches that we can take when it comes to multimodal rag and I'm going to be building a lot more content on on this. I still think RAG is one of the most practical applications of LLM and that is being used in industry more widely than compared to anything else. For example agents. The concept of agents is great, but it's not there yet. So if you are interested in RAG agents or anything related to LLMs, make sure to subscribe to the channel. I hope you found this video useful. Thanks for watching and as always, see you in the next one.