Goodbye Text-Based RAG, Hello Vision AI: Introducing LocalGPT Vision!

I'm excited about sharing this one with the community. It's my own project that I'm calling Local GPT Vision. It's an extension of Local GPT, a text based end to end retrieval augmented generation system for chatting with your documents. As the name suggests, Local GPT Vision is an end to end vision based rack system. So why vision based rag? To understand this, let's look at a quick example. Here's a page from a report on climate change. This report contains images with embedded information and tables. If you were to use a text based RAC system, it will probably fail in extracting information from these pages. But a vision based system is not going to have any problem with it. So here is an example query what percent of water is reused or recycled? And the answer is 45.6%. And here is the relevant page on which this information is found. Here's another page that the system found which is conveying exactly the same information. Now in theory you can use a text based RAC system to chat with your documents, but in that case you will have to parse the tabular and image information from your documents. You will need to do some sort of OCR along with the layout detection. Then you need to pass your tables and images to a Vision language model to generate text description and then use an embedding model to capture those text description along with the text of chunks. But what if you can do all of this with Vision language models? That's exactly what Local GPT Vision is supposed to do. Local GPT vision utilizes the Callfalli architecture which takes images of the pages, converts it into patches, then uses a vision encoder to project that into the embedding space of the LLM which creates a secondary projection of the descriptions. And then you can do retrieval on top of these projections. So here's how a text based retrieval augmented generation system looks like. You will have to chunk your documents, then compute embeddings. For chunking the documents, you need to determine what type of chunking strategy you will use, whether you want to use semantics, chunking, recursive chunking, and so on. Then you will need to determine which type of embedding model to use, what is going to be the dimension. You will need to create a vector store, then run the user query through the same embedding model, retrieve the relevant chunks and pass on those chunks plus the original user query to an LLM to generate final responses. The vision based retrieval augmented generation system makes it extremely simple. You take your documents, convert it into individual pages, then Use Call Poly to create an index of each of these pages. When the user query comes in, you do retrieval on that index. This will give you the most relevant pages based on the visual information contained in the images. This is the step at which Call Poly stops. But then you can take those relevant pages, feed those to a Vision Language model along with the original user query to generate final responses. The goal of local GPT is to make this integration extremely simple. There are two sets of models. One is for retrieval and the second group is for generation for retrieval. Currently there are three different models, although I recommend to use the Colquin 2 which is based on the Quin Vision language model is the best retrieval model at the moment. For the generation we have a lot more options. There are both proprietary and open weight models such as Llama, Pixtral Malmo and the Quinn Vision language model. You can also use API providers such as Google Gemini GPT4O through OpenAI API. Or you can use the llama model through Groq API. Under the hood, local GPT is using the Bald package for interacting with Call Poly models for retrieval. For the generation part, there's custom code for interacting with each of the models listed in Local GPT Vision. Initially, Local GPT Vision was just a branch of Local GPT. But now I made it into its own repo. So make sure to give it a star. If you like the project more exciting updates are coming to it. Okay, so if you want to get started, here is a quick description. It supports both Colquin and Call Poly models. Currently it has support for six different generation models. If you like to contribute to integrating another model, either to the retrieval model or the generation, contributions are highly welcomed. Now in order to set this up we just need to clone the repo, then create a CONDA virtual environment, install all the requirements. If you want to use one of the API providers, you will need to set up the corresponding API keys. After that you just need to run the App Py file. If you're using the CallQuint 2 model for the retrieval step, it will download the model for you. It's based on the QIN2 billion vision language model. It's a relatively small model. Next I'm going to walk you through a step by step process of how to set this up. This then we are going to look at some example use cases and later in the video I will also walk you through some limitations and failure points. The first step is to clone the repo. So we're going to copy this command. I'll paste it here in the terminal. I want to create a separate folder, so let's call it Vision based rag. This is the folder where all the code is going to be stored when we run this. Now we can CD into this Vision Rack folder. If I type ls, this shows all the files that are in the repo. Next, we need to create a virtual environment. So conduct create dash n Then let's call it Vision rag and I'm going to use Python 3.10. This is going to create a new virtual environment. I like to use conda, but based on your preferences you can use any virtual environment manager. Then we need to activate that virtual environment Conda, activate Vision rag and we are in the new virtual environment. Now, if I type ls, you can see there is a requirements TXT file. We need to install all the requirements. So we're going to use pip install requirements txt. This is going to install everything. Now, there is one small change that we will need to make and that is the Transformer package. The QUIN model needs the dev version of the Transformer package, so we'll need to uninstall the version that is being installed in the requirements and then reinstall the dev version. This may or may not be needed. Depending on the current updates and the current version of Transformer package, the installation is complete. Just one note, if you're running this on Mac os, you will not be able to install the Mistral inference package because it needs Xformer, which is not available for macOS. If you run into any issues, comment this out and it should work. Next, we're going to uninstall the Transformer package and reinstall the dev version. It's asking me to uninstall the Transformer package, so I'm going to say yes. It will install the latest dev version, which I believe is 4.46. I open the folder is how the structure is supposed to look like. We'll need to make one more change and that is going to be adding a env file. If you want to use any of the API providers, I'm going to update the repo so that it comes with example env. But depending on which API provider you you are using, make sure to provide the corresponding API key. Next we need to run App PI. To run this, we're going to type Python App py hit enter. For the first time, it's going to take some time to load. Once it loads, we should be able to access our ui. Now, if you see this warning on Mac os, don't Be alarmed. This is coming from Vllm, which currently does not support Mac os. So now we need to access our web app on this URL which is our localhost@port 5050 for the first time when you run the app, this is the UI you're going to see. In the models you have three different options. For the indexing or retrieval models I recommend to use the call Quint 2. This is the best model currently available for image based retrieval. For the generation part you can select large number of different models. So the QIN2 llama vision, Pixtral Malmo these are the models that we're going to run locally. If you want to use the API, we have support for Gemini GPT4 and for the Llama Vision using Groq to start a chat, a chat session you just need to upload your files. So here I'm going to click on this button. This is going to open another window where you can select multiple files to upload. To get started, we are going to look at some invoices. These are just dummy data that I'm going to use to create our initial index. So here are all the invoices that I'm selecting. When you click on this, then it will show you this pop up, click on Start indexing. Depending on how many files you select and how large each files are, this might take some time. Currently the local GPT vision only supports PDF files, but you can convert any file into a PDF file. On the back end you can see the indexing process happening. Okay, so the indexing is done. You're going to see a message like this. I'm going to click ok and now we can start asking questions. Let's ask about, I think this one. So here is a invoice from aws. Here are the the actual periods for which it was invoiced. The total amount is here. Right? Let's see if we can extract this data from the invoice using local GPT vision. At the moment I'm using OpenAI GPT4. I'm going to also show you these other models. So let's start with GPT4. What were the total charges on the AWS invoice? This is going to run the retrieval and then the generation pod. So it came up with the answer. The total charges on the provided AWS Invoice amount is 4.$11. If we look here, that is basically the exact amount. Now you will see that it returns three different files or pages by default. The retrieval part is going to provide Three different pages, but depending on the model you use for generation. Some of these can only process a single page. For example, the local models, except the Quin model can only process one page at a time. But the Gemini GPT4O and the Quinn model can process multiple images. The reason for returning multiple pages is that in some cases there might be information contained across multiple pages. At the moment this is limited to three, but you can adjust it based on your own needs. Now the good part is that once you index, you can change the model first generation based on your own preferences in the same session. So for example, we are going to ask the same question, but this time we are going to use the Gemini model and let's see if there is any changes in the response that we get. Okay, so the Gemini provided a lot more details. It talks about the actual invoice amount which is correct since it was passed on multiple different pages. It also looked at the the invoice on this page as well as on this page and that's why you see the invoice amounts for these other companies. Now let's see what happens if we use one of the local models. I'm going to use the QIN2 Vision language model which is the 7 billion instruct. For the same prompt on the backend it's loading a checkpoint. In my case it has already downloaded, so it's going to be relatively quick. But if you run for the first time, it has to download the model and then run it. At the moment for the local models I'm using 16 bit floating point precision because I haven't seen a good implementation for the quantized versions of these models. Here's the response that we get from Quinn. I don't know if it's like 4 point or it's like 4111. The response really depends on the quality of image you're providing. Depending on the actual image or document quality plus the resolution, the response might be a little off. I'd highly recommend to test out different models because local GPT Vision provides multiple options. For the rest of the video, we are going to just use the Gemini model because that is the fastest among the models that are here currently. The Groq Llama vision model, the 8 billion llama 3.2 model has some issues with the implementation that I have, so usually I don't really get good responses, although this speed is phenomenal. So I'm going to look at this and try to fix it. Another point is that the Llama vision is also llama 3.2.8 billion model. More details are in the readme of the GitHub repo. Okay, in order to add another chat session, just click on this plus sign. Then you will be able to load more files and start interacting with the models right away. You can easily switch between different chat sessions that you have. For example, here is a chat session related to one of my trips. You can continue the conversations. The way I'm doing this is that with each session there is a associated session ID and then it loads the index if you switch between sessions. It's a little memory intensive at the moment, so I think it does need a lot of optimizations. It's a work in progress just to show you a couple of failure points. For example, I asked it where is the IBIN of this company? It was able to give me the correct company document or page but it messed up some of the numbers. I think for this specific one I used the Quinn model. Now you will see this image settings. This only applies to the Quinn models. It doesn't impact The Gemini and GPT4O or the other models. The reason is that the Quinn model supports dynamic sizing, so you can resize this the way you want. So I'll highly recommend to increase the image resolution if you're using the Quinn model. But also keep in mind that will increase the VRAM requirements. Now the great thing about this vision based approach is you can ask something like can you explain figure 14? If you look here, it actually retrieved a page that contains figure 14 and then it talks about everything that is in that specific diagram or figure and I think it does a pretty decent job. For example, when I asked about instruction tuning, it refer to a page that talks about instruction tuning. Now another test that I ran was I said extract table 10 and return it in markdown format. Here is table 10. Depending on the LLM you choose, the results can be hit or miss. I think this was Gemini 4 which got most of the numbers correct. But when I switched to this was I think the llama 3.28 billion. So it doesn't really work at all. It just made up a whole bunch of stuff. Here are the results from GPT4O but the layout is not correct. In this case it's using a very naive system prompt. So I think I can easily improve on the system prompt and hopefully that the extraction results are going to be more accurate. Anyways, with the vision based rag you can ask complex questions especially on images and tables. That was not possible with the text based RAG systems. Or at least you will have to go through a number of different steps before you can even parse those images and tables. So one of the example is here. I actually asked it to elaborate on the hotel lobby and there's a picture in here and it's able to describe that picture without any issues. Okay, so this was a quick introduction to local GPT vision. If you like the project, make sure to give it a star on GitHub. I'll really appreciate that. If you like to contribute, check out submit a pr. I'll be very happy to merge your contributions. I see a lot of potential in vision based drag, especially for information extraction and end to end rack systems. If that interests you, make sure to subscribe to the channel so that you're up to date on all that is happening in this particular space. Anyways, I hope you found this video useful. Thanks for watching and as always see you in the next one.