ColPali: Vision-Based RAG System For Complex Documents

Retrieval augmented generation or RAG or documents that contain tables, images and text is a complex task with its own challenges.

In this video I will introduce you.

To a new technique that addresses some of these challenges.

Here is how a standard RAG pipeline looks like.

With a multimodal RAG system, we first need to do some sort of OCR and layout detection on our documents that contain images and text. Then we need to parse images and tables from the document and use a vision model to generate description and captions for text. We need to decide on a chunking strategy to split text into chunks. And then we need to use an embedding model to generate embedding representations for these text chunks as well as for our captions or descriptions. During the retrieval process, we are doing similarity search based on the embedding representation of text chunks. As you can see, there are a lot of moving parts and each one of them will introduce its own challenges and potentially introduce errors in the pipeline. Now an alternative is to just directly embed images of these documents. In one of my previous videos we looked at Call Poly, which is a new technique that basically converts each page in your document into an image and then use a vision language model to encode the information contained in this image into a vector representation. And then during a retrieval you encode your text query and do a very similar similarity search. But in this case you are doing a multi vector representation rather than using a single dense embedding vector.

And the great part is you actually.

Get get a lot more explainability.

So the same model can be looking.

At text as well as images and if it decides to use images or text can actually tell you where exactly the information is coming from. So in this approach you only have to embed your documents as images and you don't really need to do the.

Rest of the pre processing steps.

Now I believe the Gemini API does something very similar. Whenever we upload PDF files, you don't really have to parse PDF files. The Gemini API does everything on the backend and that's why I'm a really big fan of this specific feature. But you can replicate most of this.

Functionality locally thanks to a couple of new Python packages. So in this video we're going to.

Build an end to end system where you just need to upload a PDF file and you will be able to chat with your PDF files. Now here is how the system is going to look like. We will take a PDF file, then we'll use the call poly model to generate a multi vector index. It's going to be Using Latent direction based on Colbert style models.

So in fact we're going to be.

Just encoding each of the pages of the document as images and storing the multi vector representations of those images. Now, during the user query we will use the user query, embed them using the same Colbert style model and then we will retrieve documents which are going to be specific pages which contains the information requested by the user. Next, we pass on those images along.

With the original user query to a vision language model.

I'll show you how to do this using CLAUDE as well as a local Quinn model. So you'll have both the options if.

You want to run this locally or.

If you want to use an external API. We'll pass on the user query plus the pages which are relevant to the information requested by the user to generate a final response.

So this is going to be a fun video and you will be able.

To build an end to end multimodal vision based Rack system. So let's get started.

Here's the notebook that we're going to be using. This is based on two other fantastic notebooks. One is this multimodal RAG using call Poly and Quin2Vision language model. The second one is chat with your PDF using Baldi Claude. Both of them uses this new package called Baldi which enables you to use Call Poly in a very easy fashion. Varldi was created by the creator of ragatui, which is another fantastic library that you definitely want to check out for building rag pipelines. It enables you to use late interaction approach used by Colbert. This is a multi vector representation instead of a single dense vector representation used by most of the current RAC pipelines. I have created quite a few videos on this. I'll put link to some of those in the video description. Now this new package is under active development. It's actually a pre release version. So there could be a lot of changes in the future. But I'll highly recommend to check out the documentation in this video. My focus is going to be to show you how to build an end to end system which will use Call Poly for for converting multimodal documents into image representations and then compute their multi vector representation which will be stored as an index. And then we're going to use a vision language model to answer questions based on those documents. So there are going to be two parts. In the first part we will use CLAUDE as our vision language model. And in the second part we will use Quinto Vision language model if you want to build an end to end local system. So let me walk you through the setup. First we need to install the required packages. So this will include BLD and Claudette. Claudette is another package which enables you to interact with Claude models. It's coming out of ANSA AI which is run by Jeremy Howard. Jeremy is known for his work on fast AI. He's one of the best educators and whatever he builds is worth checking out. Next you want to install the latest version of Transformer package, plus some Quin utilities. You also want to use Flash, Attention and Bits and Bytes. If you are running this on an Nvidia gpu, we'll need to set up our API keys. So I'm using the hugging face token because we need to download Call Poly. For that you actually need the hugging face token. And in the first part of the video I'm going to be using Claude as a multimodal model or the Vision language model. So we'll need that API key as well. But you can run everything locally if you want. So you don't really need this step. But keep in mind if you want to run this locally, you will need high RAM GPU. You cannot run this on a T4 GPU. Next we will import rag multimodal model from Balyadi. Okay, so first we need to download our model. So as I said, we'll need to download the Call Poly model. This is going to take about four to five minutes depending on your connections. For our example documents we're going to be using this. Attention is all you need paper which contains images, tables and text. So here we are downloading that paper and putting it into another folder called Docs with a new name called Attention PDF. The paper has 15 pages. So next we run the indexing process. So we need to provide the path of our document, then the name of the index. This can be anything that you want. And then you also want to tell the model whether you want to store the base 64 representation of each of the pages. So basically it will convert each page into an image and will store the base 64 representation. You don't need this. You can actually just create multi vector representation or embeddings of each page and store those. And then when you retrieve a certain page, you can convert that into base 64. I'll show you that process later in the video. But by default we will enable storing of the pages as base 64 as well. And if you run this multiple times, we'll just overwrite the previous values. Right. So when you run this specific cell, you will see that it will create indices for each of the page and store them. Now you can just provide a path of a folder that contains multiple PDF files and images and it will do the same process for those. Okay, now something to keep in mind if you want to use Colbert style multi vector representation. Most of the current vector stores that we have, they don't really support multi vector representations. So you will just need to store this directly on disk. I'm aware of VISPA is one of the few vector stores that actually supports multi vector representation or Colbert style representations. They also have support for Call Poly, but I don't think any of the other vector stores have this support yet. So this is something to keep in mind if you are thinking of putting something like this in production. Okay, so once we index our documents, the rest of the process is extremely simple. So here we have our query. Then we run this query over our index and the result is going to be something like this. So the result is the page number, what is the score and then base 64 representation. In this case we are asking model to only return a single page. If the information is contained in multiple pages. You can actually tell it to return like four or five different pages. But essentially this is going to be the representation of a single page in base 64. So our query was what is the blue score for the transformer base model? This information is contained in a table on page eight of the Attention is all you need paper and here are the scores. Now you can see that it's going to be extremely hard for any RAG model to retrieve this information because it's contained in a table and then there is right next to it another column or another row that has some other values for transformers. But we are specifically asking it to retrieve the the information for base model. Based on our results, this information is contained on image 8 or page 8. So we convert this base 64 representation to image data and we display that image. So here's the actual image of the page 8. You can see that it was able to retrieve the correct page. So so far this is what call Poly will help you. It will only tell you where this information is contained. Now we need to pass this information to a Vision language model or a multimodal model along with our query to generate our final response. And in order to do that, we're going to be using the Claudette package. This enables us to interact with CLAUDE models. So for example, we have a list of different models. We're going to be using the Claude 3.5 sonnet.

So that's a chat model.

We pass on our base 64 representation of the page plus the query. And here is the response generated by the model. There are two numbers. One is translation from English to German and English to French. So these are the two numbers that we are looking at and you can see that the model is able to correctly retrieve that information. We can look at some more details. So it says according to the table in the image. So it's actually specifically looking at that table. It was able to retrieve the proper information for us. Now one thing that you need to keep in mind is the number of input tokens. So since it's processing images, it is going to use a lot of tokens. In some cases you want to down sample the images. So instead of high resolution images, you probably want to reduce the resolution. That way you will be able to save on these tokens. But keep in mind it could impact the accuracy of the responses that you get. Now, even though the number of tokens used might be a bit high, but I think it saves you a lot of headache in terms of the RAC pipeline that you need to create for parsing the data, then computing embeddings for different chunks and storing those and then ensuring that the retrieval based on those chunk of text chunks is accurate. In this case we're just directly encoding pages as images and then doing retrieval on those. Next let's look at how to use completely local system. So the call poly based retrieval is local, but for the vision language model we were using claude, but let's replace this with a completely local solution. Now in this case we're going to be using the QIN2 Vision language model.

And I'm using the 2 billion instruct.

Version which is currently the best Vision language model for its size. We download this model, put it on the gpu. We also download the processor which is essentially a tokenizer for Vision language model. I also want to show you an alternative approach. Here we're storing a base 64 representation of our documents. So each page is converted into an image and we are storing that base 64 representation as a part of the index. But you can skip this step because it's already computing the multi vector representation that is going to be used during the retrieval step. So in here I'm converting each of the page into an image separately and it's not being stored as a part of the index itself. You can see this is page number eight. Then based on the results for our query, we get the index number. So it simply tells us where to find that image. Now the index image is 7 because indices in Python starts at 0, so this still points to image number 8. Next we're going to take our query and create a message for our model. So the message is going to contain the image plus the initial user query and we will pass this on to the processor. So this applies the prompt template for this specific model and then here we are processing it to convert it into different tokens. Now we run this through the model and here's the final output. So it says the blue score for the transformer based model is 27.3, which is correct if we look at the English to German translation. Now since it's a much smaller model, it missed out on the second part, but if you use the 7 billion version of QIN2, it probably is going to be able to tell you this part as well. So in general, if you have a.

Multimodal rag system that you want to.

Build, my recommendation would be to explore this vision based rag rather than a.

Multi step multimodal rag based on an LLM.

And if you're looking for help, you can reach out to me. A link is going to be in the video description. I do offer consulting and advising services for startups. I hope you found this video useful. Thanks for watching and as always, see you in the next one.