Can VISION Language Models Solve RAG? Introducing localGPT-Vision

It's been a while since there was any update to Local GPT, but I'm really excited to share this one. This new version is called Local GPT Vision. It comes with a completely new UI which can not only give you the answers, but directly gives you the pages where the answers are coming from. Like the previous version, everything is running locally, but you also have the option to use external APIs and now everything is powered by Local Vision language models. Everything is still hundred percent private and nothing leaves your system. Let me first walk you through what is different in this version of Local GPT. Then I'll tell you how you can set this up on your local machine and in the last part of the video I'll talk about some tips and tricks if you are using this specific version. Now this version is available on a separate branch called Local GPT Vision, so it's not available on the main branch yet. And if you're not familiar with Local GPT, it's my own project that lets you chat with your documents securely locally using open source models. So let's first change the branch so that we can go to the Local GPT Vision. This is an end to end vision based retrieval augmented generation system. It allows users to upload and index documents. Currently it supports PDFs and images, you can ask questions and you will receive responses along with the document pages where the responses are contained. And it is an end to end vision based system. So there is no dense embedding model, you don't really need to do any chunking, anything like that at all. Everything is vision based. Now it's a two step process. In the first step we retrieve the most relevant pages in the documents. That is done through Call Poly. And Call Poly is a new technique which uses Vision encoder to find information in documents. I have a couple of videos on this topic, will highly recommend to check those out. And the next step is to use a Vision model to retrieve information or generate a response for that. The project currently supports three different models. One is QIN2 Vision language model, the other one is Gemini and the Last 1 is OpenAI GPT4 and we are able to do all of this thanks to this amazing library called Biald which makes working with Call Poly extremely simple in terms of features. It's an end to end vision based rack. You can upload documents in and index them. There's a chat interface now you can have multiple sessions so you can have multiple different charts. And as I said it currently supports three different models. Although primarily the focus is on the local and open source Quinn2Vision language model and you can persist indices. There is a small bug in here that's going to be fixed pretty soon. The first question is going to be why use a Vision Language model? This is an image from the Kalpali paper, which really explains the need of using a Vision Language model. So in standard drag pipeline, which local GPT is based on, you usually have information contained on images. Then you normally do some sort of OCR or layout detection, or you extract text from these pages using libraries like unstructured I O or Llama parser. Then you need to determine what type of chunking strategy you're going to use. There are a whole bunch of options which are going to impact the performance. After that you compute embeddings for those chunks. Now, again, the choice of embedding model really impacts the performance of your retrieval accuracy. So all of these things add to the complexity of the system as well as it and adds to the potential errors in the system. On the other hand, these multimodal models are getting really good. So Call Poly proposes using a Vision encoder to directly encode the information contained in a PDF file or images. And then you can combine that with a language model to directly do retrieval. Now, in case of Call Poly, you can only retrieve the most relevant pages. The model is not able to generate final answers. You need to have a second stage of generation. So for that we basically use the relevant pages that are returned by Call Poly stage to feed it into a Vision Language model along with the initial user query to generate final responses. And these models generate responses by understanding both the visual and textual contents of the documents. So it's a very powerful approach. If you have worked with something like Gemini or GPT4O, they are very good at information retrieval from images. The quality of the response is highly dependent on the model that you use as well as the resolution of the document. And I'll address this point later in the video before looking at the setup. Here is the architectural diagram of the whole system. So there are multiple different components. One is the user facing ui. Then in backend everything is running in a flask app. We use Call Poly for creating indexes of the documents that the user provide and we save them on on disk. So you're going to see the actual documents that are uploaded, indices and then all the information related to sessions. The retrieval module is using Call Poly to retrieve relevant images to the user question when the users ask a question and then those are passed on to the Vision Language model to Generate the final responses. If there is interest. I'll create a more detailed video on covering this architecture in the code. So in terms of setup, you want to use some sort of virtual environment. I like to use Conda. You will need a Python 3.10 or higher. And you also need Git if you want to clone the repo before the installation. We are close to 20,000 stars, so if you like this project, make sure to give it a star on GitHub. Let's talk about the installation. You want to clone the repo with this local GPT Vision branch, because the newest Vision version is contained in this branch only. So let me show you how to set this up. I'm using a virtual machine provided by Mast Compute. There's a link in the video description if you want to use the same machine, are going to get 50% off and I'll get a small commission out of it. So first we're going to copy that command and this, then clone the repo. Then I'm going to type look CD local GPT vision. This is the folder that we created. Next. If I type ls, this shows us the contents of the folder. Okay, so I open this folder. Next we need to create a CONDA environment. We're going to use conda, create dash n and then let's call it local. Then I'm going to be using Python 3.10. So this will create a new virtual environment for us. In my case, the virtual environment already exists, but let me remove this and recreate the whole virtual environment. Next we will just activate the virtual environment that we just created. And then we're going to use this virtual environment to install all our required packages. First let me install all the required packages using PIP install R and then requirements txt. This will install everything that we need in terms of the required packages. We have flask, baldi, torch, torchvision, and if you want to use the OpenAI or Gemini model, you also want to install these two packages. But if you want to use the local QUIN model, then you need to install the QUIN Vision language model utilities. We also need to install the transformers package. So if you run the requirements txt, it will install the transformers package, but you want to uninstall that version and install the dev version. So first we are going to do pip uninstall transformers because we want to install or reinstall the dev version. So this will uninstall the transformer version. Next we're going to copy this command, which is basically the latest dev version of the package and we're going to install that. Okay, so the installation is done. Next, in order to run the ui, all you need to do is just run this app PY file. We're going to type Python app PI and run this. Okay, so our UI is up and running on localhost port 5050. So here is how the UI looks like you. You can create new chat sessions here. You will be able to upload all your documents here and then you can create a chat with the model in here. So I'll walk you through this, but first let's go to Settings. In Settings you have three different options. Currently the Quinto is working. I think I still need to update the code for Gemini and GPT4O. I'm going to also be including the anthropic model as well. Claude 3.5 sonnet. Now you can provide the image resolution here. They need to be multiple of 28 and this will determine both the quality of output from your second generation stage as well as the memory requirement that you're going to have. By default we're using the 16 bit precision of Quinto, but you can use the quantized version if you want. You can also use the 2 billion instruct version as well. Okay, so let me walk you through a quick example. I'll be creating more detailed videos on how to use this. So first click on Browse. We're going to select one paper first and then later on I'll show you how to select multiple files. Currently this works only with PDFs and images. I'm also adding support for docs and HTMLs, but in that case everything is going to be converted to to convert it into PDFs and then process. So first you can select one or multiple files and then just click on Upload an index. This will start the indexing process. So first it converts the pages into images and then use the call Poly model to create indices. Now if you are running this for the first time, it's going to take a while because it has to download the models and then do the index and process on top of that. But here you can see that the file contains I think around 27 different images or yeah, 27 images. It converts each one of them and create these indices for us. Now if you go to the upload Documents section, you can see that the paper was uploaded here. There is a session which keeps track of all the user conversation with the model. So that happens here and then the actual index is created with. Within this folder we can start With a very simple prompt, who are the authors of this paper? I'm going to say give me the list and then send this on the back end. You can see it's loading the model now since it's using the call poly model. So it first run through that model and then it uses the QUIN model to generate the final response. And in terms of the response you can see it says the authors of the paper are Nicholas. And you can also see the pages that it has returned. So by default it will return three different pages. Here it picked the first page because it contains the name of the authors. In general question like this can trick some of the standard RAC systems because it might see a lot of author names in references and I've seen that a lot of time it will return or confuse the references author list with the actual authors of the paper. Here's a more specific query. Does the paper shows a drop for the H value in the layer norm? So I took this from here in the paper and according to the paper it does show a drop. But the interesting thing is that the information is only contained in this page. And the response that you get is it says yes, the paper shows the drop in the H value in the norm, but the first page that it returns is actually the page where the information is coming from. Now the good thing is, since it's a multimodal model and you are using a vision language model on top of it, you can actually directly interact with images contained in a paper. So you don't really need to parse these images separately and then give them to a separate multimodal model. The vision language model can actually understand these images directly. Okay, so a couple of things that we can do with this. We can click on New chat. You can also edit the name of the chat session, but I'm going to upload two files this time. So it will contain both the paper that we are looking at stealing part of our production language model, and also the original ORCA paper. And then click on upload an index so it will start indexing the documents again. Here you can see that it's uploading the first page or the first paper. And then it will index the pages and then after that it will index the second paper for us. Okay, so it indexed two files. So you're going to see a list of files here. And let's edit this to, let's say we're going to call it Papers and save the name. So you can see that the name has changed here. Now the second paper that I uploaded is this one. So here's an image that talks about the performance of different models on zero shot reasoning tasks in Big Bang Bench Hard task. Okay, so let's ask a question related to this image and let's see what type of responses we get. So my question was which model has the best performance on zero shot reasoning tasks in this big bench benchmark? And you can see that it actually got the same page on which we have that image. And the response is orca13b. It does a pretty decent job at retrieval. Now the accuracy of retrieval really depends on the quality quality of the models that you're using as well as the resolution of the images. So for example, here we can go back and let's say potentially use a very small image. So I'm going to save this and I can continue asking questions here. So let's run the same question again and let's see if we can get the same response with a smaller image. Now in this case for the same question the answer is chat dpt. It's again retrieving this the same document or same image? Because the first part doesn't really depend on the resolution, but the second part is the Vision language model which does depend on the resolution of your input image. So if you reduce the resolution even though the proper image was returned or the proper page in the document was returned, still the model has trouble figuring out the information. And this is one of the downside of using the Vision language model. Now this mainly works on PDFs and images, but as I said, you can convert other document formats into PDFs and they will work fine. There is actually a code which will convert any word documents into PDFs before processing them. But in general I do see a lot of value in using completely vision based retrieval augmented generation, especially for documents. You can potentially use this for CSVs and Excel files as well. But in that case I think you will need to add the header in every page that you're retrieving. So this was one major update. Again, it's on a completely separate branch. There are other updates coming to the original local GPT as well. In that case we are going to be adding a lot of different things such as re ranking, query augmentation and so on and so forth. So you will definitely see a huge performance improvement. If you are interested in incorporating local GPT or other vision based RAG systems in your own pipelines, you can reach out to me. Details are in the video description. Also, please consider giving it a star. Let's go through that 20,000 mark. Apart from local GPT, I have also a couple of other open source projects. One is called Verbi, which is a voice assistant. This uses both proprietary as well as open and local models and the other one is Agent Zero which is trying to replicate one's chain of thought reasoning. If you haven't seen those projects, I highly recommend to check them out. Make sure to subscribe to the channel as well. I hope you found this video useful. Thanks for watching and as always, see you in the next one.