How to build a Streamlit UI for Local PDF RAG [Ollama models]

In this video, I'm going to show you how to build a Streamlit UI for the RAG PDF application that we built in a previous video. The RAG PDF itself runs locally on your system using Ollama models. And so you would need a user interface for you to interact with that application offline. So Streamlit is an open source Python framework that allows you to build very dynamic data applications or AI applications really in this case, to be able to interact with your models. And your apps. So let's get to it. All right. So let's go over the flow diagram first of all. And maybe you've seen this from the previous video, but I just want to do like a TLDR of what we are implementing or what we are adding into this flow chart you can see here. So in this flow chart, initially, this is our entire RAG pipeline with also the RAG application. And this is the new portion here with the Streamlit portion here that we didn't have in the previous app or in the previous implementation. And so The previous implementation here, we had PDFs, which would be files that you've downloaded in your system or you have them locally already. And then you're passing it to the LangChain application that we built, which was unstructured PDF loader as the function. And it does take those PDFs and then parses out the entire PDF and extracts all the text within the PDF. At this point, I haven't experimented with it grabbing images from PDF, things like images within the PDFs or tables. I haven't added the full functionality to be able to parse that correctly. Maybe that's for another video, but for this one it does parse text pretty well. And this is all LangChain modules that you can use. And once it's parsed the text itself from all the PDF documents, it then recursively chunks the characters within the text that it's downloaded. So it will use maybe 7,500 window or whatever window you set. And so it will, every, after every 7,500 words, it's gonna cut or split it. And that will be the first chunk and then so forth and so on until it gets to the last last character within your document. And all those chunks are then going to get passed over to an embedding model. In this case, it will be Nomic Embed, which is served by Ollama, downloaded from Ollama. And it will create vector embeddings off of those texts or chunks and then pass those vector embeddings to a vector database, which essentially stores all the vector embeddings that we've generated using our vector embedding. And once it's already stored in the vector database, which in this case is Chroma, and you can swap that for maybe another vector database if you don't like Chroma, it could be something like Quadron, Milvus. The list is really endless at this point. And then once that's stored, then we go back to the user. Now the user is, which is you as the user, you put in a query now to chat with that PDF, right? You tell it, hey, maybe summarize this, or what is the main concept? What are the takeaways from this entire document? And so you'll write that query on a Streamlit app. So this is the kind of like the face of the— Actual application. So you'll type in on an input box, hey, what are the main takeaways from this file? And once you hit enter or submit, then Scrimmet would pass on that question over to the multi-query retriever, which is also a LangChain function that then will go ahead and generate. In this case, I set it to 5 last time. This time I set it to 3. So it will generate 3 similar questions to the one that you've entered. Or if you want 5, if you want 10, you can, you know, You, you can set it that way. It's just a prompt. But the caveat to that that I wanna point out to you is the more questions you put as more of like to generate different variations of it, each and every one of those questions have to be embedded. First of all, your embeddings of it will be created of that question, and then it'll be passed over to the vector database, which is Chroma here. And since the material that you already passed into the vector database are in vector space, your question also, which is now a vector, will be kind of try to match which is the closest semantically to another vector within that vector space, and then it will retrieve out as a context. So if you have more questions, it will exponentially kind of increase the amount of time that it has to go and return those contexts within their vector embedding. So I put 3 because it's relatively fast, but it's still slow compared to 1 question. But then once it retrieves those contexts, we'll all— from the example for -3 questions, it's going to return 3 contexts. After it returns it, it will then get unionized or create a union of all those 3 questions plus the original question that you asked. And the contexts from all those 3 will kind of get, you know, one final context that then enriched context in this way. And then that context that you will get from that plus your original question, which was your prompt, will then be passed over to Mistral LLM that we've already selected. You can pick any LLM really, so long as you've downloaded that on your system. And that context and prompt will be passed to the, to the model. The model will then generate a final response, which is more coherent, which then will be displayed on the Streamlit app and you'll be able to read it and you're happy or not. But essentially here, what I'm saying is the implementation that we will do will be this portion, which is the Streamlit portion, which is really where you enter the con— the question. And then things happen in the background, which you already went deeply last time, and I'll show you a little bit of it as well today. And then once the response comes back, it also just throws us back to the app so you can see So instead of like looking at the terminal and trying to read through it or Jupyter Notebook, Streamlit is an easy way to do it locally and you can customize it the way you want it to look. So without further ado, let's just go ahead and look at the app and the code as well. All right. So let's look at the app itself first, the final product of what we will be building or what we'll be implementing in this case, before we look at the code and we'll do it side by side afterwards as well. So you can see what does what and what it looks like on the app itself. And so if you look at the app here, this is the Streamlit app itself. And the first thing at the top here, first of all, I have a brain and some text here. And this essentially is a subheader. I could have put it as a title, but this looks much cleaner and easier. And I don't want a big chunky text up here for nothing really. And then I have a divider, which kind of divides the content here to make it just look aesthetically okay. You can fix that however you want in your app itself. But The main functionality of this app, essentially it allows you as a user to upload a PDF with a limit of 1 gigabyte for the entire file itself for the PDF. And this is very configurable in Streamlit under the Streamlit configurations file. You can configure this to, I mean, it does come with a 200 megabyte default size here, but you can adjust it to however size you need for your files. And you can also set like the specific file type that you want this widget here for uploading files to be able to import. So in this I'm putting it as PDF. So if you try to import like an Excel file or a CSV, it's gonna, it's not even gonna allow you to do it just because I've set it as type has to be PDF there. And so I've already uploaded a file here. It's an AI agent security PDF file. And this file, I got it from arXiv, which is kind of like an open source where people kind of post their research papers there. And it's become really popular since the LLMs got really popular. And in this case, I wanted to upload a recent paper so that way, or document in this case that I'm sure the LLM wasn't trained on, or at least some of these LLMs that I'm using were never trained on. So that way I'm able to kind of get the feel of what they're extracting from these PDFs. And if they're giving me actual context from these PDFs, or they're just kind of hallucinated from previously pre-trained material. And so in this side of the app, and all this is really aesthetics, I split the app into 2 sections. This first side allows you to upload the PDF and then it does display that PDF here in the file, which you can scroll and view it as well. What I've also noticed, like it depends on the PDF itself. Like for example, this one is not very legible. That's why I implemented this zoom here. And so you can be able to zoom. Streamlit doesn't have like inherently kind of natively built zoom-in features. So I just had to implement this with the ruler, Streamlit ruler. And it just allows me to be able to zoom into the text and try to read a little bit of it. But it also depends on the PDF. Some PDFs are like super legible. This one is not that amazing. It was really hard to read some of the text in there, but at least for the diagrams, I can see the diagrams and, you know, If I want to really, really, really focus on it, I might be able to read some of the stuff. But that aside, that's the functionality of this side. And at the bottom here, we have a delete collection button. It'll delete all the embeddings that we've created from this uploaded PDF in our vector store, which is Chroma in this case. And so once this part is done, we have this other column as well on the right side. And this first section here allows you to pick a model that you've already installed on your system using Ollama. And so I have Quen. For B here, I have Mistral, Llama 3, and then I have this embedding model for Nomic Embed. And if I delete some of them, of course they will disappear from here as well, because it's just checking our system to see what we have and then automatically showing them here as options for us to use. So I'm going to use Mistral because for some reason I've found that it does generate better responses for me. You might find some of those work better for you. And also the reason why, as you can see here, I've already ran some of this model. I've already chatted with this document a little bit. I've already asked 2 questions because I tried doing this earlier and it happened in the previous video as well. I don't have a lot of good memory system in my— I don't have enough compute in my system to be able to run this while also recording this video. So what I've seen is whenever I run this model, whenever I run the prompt, they compete for resources and my video suffers. It just kind of kills my video. So In order to avoid killing my video so I can record this for you, I just pre-ran some of it and that way I can record it and you can try it on your end as well. So here's how it looks like. And it takes a few minutes to really generate this context. For this one, I asked it, what is this document about? And it generated a response here. As you can see, it says this document discusses security vulnerabilities and defenses for AI agent systems, specifically those based on large language models. So essentially this paper is about AI agent security. And it's a very interesting paper which kind of got really released recently. So most people are talking about AI agents, but not a lot of people are talking about the security of these agents in production. So these people kind of try to address some of those shortcomings or where they feel like this could be potential loopholes and they're trying to provide solutions for that as well. So it does provide a good summary here, which I think is correct. And it says the text covers 3 types of vulnerabilities. Local vulnerabilities related to data manipulation and system availability, remote vulnerabilities allowing agents to attack external services, and planning vulnerabilities enabling denial of service attacks, which is pretty interesting to see this. And then I also had asked it in the prompt to be able to provide context snippets of where it got these responses from, from the paper. And so it did cite here some direct kind of related information, some of the context where, where we get some of this to be able to draw that conclusion up there as well. And so it's pretty interesting to see that it generated all that. And then here, the next question I asked it, why is it important to have session-aware AI agents with memory? And that's exactly something that was somewhere in here on one of the diagrams. And it didn't go into detail, but I just wanted to see if it can return some of the context that was written on that paper as well. Here, let's go through the code and I'll show you exactly what was happening in this code. And so let's start up here first before we get to the terminal anyway. And so at the top here, what we do first of all here, and maybe I can zoom in a little bit on this material. And so here on the first section here, we're importing modules that we will be able to use in this application itself. So we're importing Streamlit because this is a Streamlit app. And then we have logging, which will help us log some of the kind of what's happening within the script that we have. And then we have the temp file. We have PDFPlumber that would allow us to be able to convert the PDFs into images. So we can display it like we did in the app right here. And so we'll take the PDF, convert each page into an image, and then be able to— we'll be able to display here using Streamlit image, st.image. And then we have all the LangChain family modules that we will need to be able to build the RAG orchestration portion of it, which most of this we did in the previous video, but we'll also go over how we structure that into functions within this application itself. And then of course typing for just kind of decorating our functions to make it look more Pythonic. And so the first thing we want to do is set the page configuration. And if you're using Streamlit, you want to have— you want that to be the first line of code at the top of your application, because after the imports, of course, because if you don't do that at the top, Streamlit will throw an error and it doesn't like that. So let's go ahead and see how this works. So page title, it just does exactly what it says. So it just sets the page title and you can see up here at the top, We have the Ollama RAG Streamlit UI. That's exactly what we wrote there. And then the page icon, it does the icon right there as a balloon. And then the layout is wide. We just wanted to cover the entire page. And then the initial state of the sidebar, which we don't have a sidebar because it's just a single page app. It's not a multi-page app. It's collapsed. So it doesn't even show up there. So we don't even need to set that. And then here we are just instantiating the logging configuration. And this will just kind of show you all the information that we'll need or kind of what the steps are going on in the app. And then we're initializing the logger right there. And then the first step here, we are creating the function that extracts model names from a given model list of model information, which would be a tuple of model information that we get from Ollama list. And so once we do that here, so what we do is we pass the models information, which is all the information from the models, and then We're adding a logging here to tell us it's extracting the names of the models information here, and then the model name. So we're just kind of iterating through it, and we're extracting the name from the list of models that we got, and then just returns all the models there. And cool. And then the next thing here, we're creating the vector database itself, and we're passing in the file that got uploaded, which is the PDF, to be able to create embeddings of that. But in this case, what we're doing first of all here, we are logging, of course, the step there, the name of the file, and then we're creating a temporary directory, and we're using the tempfile module from Python. And this will create temporary files folder where you can just kind of append some of the files that we will need to be able to work with here. And then once we're done, we clean it up so that we, you know, we're saving up on memory here. So we're creating here, we're grabbing the path to that temporary file location. And we're passing in the file name that we just uploaded there as well. And once we've done that, we get that file, we use unstructured PDF loader. And we talked extensively about what this function is for. So this unstructured PDF loader, It will take that PDF and kind of get all the data that's within the PDF itself, the text, and be able to structure that in a way that we can then do great vector embeddings off of it using a vector embedding model, and then be able to save that in a vector database. And so the next step here, we are creating chunks of this text that gets scraped from this PDF itself. And we're using the recursive character splitter. And what it does, As we're creating chunks of 7,500 characters. So after 7,500 characters, it creates first chunk. And then the next, after 7,500, it creates another chunk all the way until that PDF document material is done. And so what we are doing also, we, we are adding this chunk overlap. Think of it as a glue between each chunk. So that way, if maybe the first chunk cut a sentence in the middle of it or in the middle of a word, the glue is able to glue between the previous text and the previous, the previous chunk and the next chunk to be able to make it a cohesive, narrative. So that way when you return the context, it's able to understand exactly what's happening instead of just kind of picking up from the middle there. And then so the chunks here, we kind of putting them here. We're passing that, this is using that text splitter there that we've instantiated. We pass in the data, which is all the data we, that the unstructured data is already kind of corrected. And then we're creating the chunks. And then so we've created the chunks there and then we are instantiating the embedding models here and we are using Nomic embed text, which I've already downloaded my system. It's from Nomic AI company. It's an open source embedding model, which is pretty good. And so we use that, we create an instance of that, and then we create the vector database here. And we're using Chroma in this case. And from documents, we are passing in the chunks from all the documents that we've chunked there. And then also the embedding model that we've created here, which is that Nomic embed model. And then we're also creating a collection name, which is my RAG. Think of it, if you know databases, it's like a table. Just want to have, you're going to have different collections within an entire vector database. So same, same kind of thinking there. And so here we're creating a logger, of course, to tell us the vector database has been created and it's done. And then here we're grabbing the temporary directory and we're just removing it because we don't need it. Just going to save up space here as well. So we're emptying it where we saved the PDF file initially, temporarily. And the next function here is process question and it does what it says. So we process a question using the vector database and selected large language model. And so we'll take in your user question. So once you write in a question, the logger here is gonna return the question and also the selected model that we have. And also we pass in the vector database, which contains the document embeddings and then the name of the selected language model that you picked here, which for us is Mistral. The first thing here, of course, the LLM model here, we're passing that select LLM model. And here is where you do selection in the app. So this is where you have this selection of models. You pick one that gets passed in there as well as the model itself. And then the temperature for that model, for example, here Mistral, we set it to 0. And temperature kind of just determines of how the model is going to be really creative. The higher the number, the more creative the model is. 0 is kind of, we're trying to make it more repetitive in this case as well. And we create here the query prompt and prompt template variable there. And so the prompt template, we are passing in the input variable to be a question. And then here's generating, we wanted to generate 3 similar questions from our initial question. So the original question you ask, it then gets passed over to an LLM, which generates 3 similar questions from that question that you created. And that way it'll be able to help us generate better context from the vector database itself. So if you think of it, you have a box, you have different dots within that box itself. You just maybe in this case, throw a dart. And this is just a very rudimentary example that I'm giving you. It might not even be the best one, but you know, hang on with me. So you throw a dart within that box inside and maybe you see so many dots, you throw one dart. There's a chance it's gonna hit the target, the right target, which is really the context that you're looking for that question. It's going to embed and get closer to the question that you, the context that you, that's related to that question. But sometimes your prompt might not, or your question might not be worded correctly or best way worded correctly to get close to the answer. And so if you tell the LLM, take this question that I've generated, generate 3 similar questions that kind of matches the same context. And so it will go and generate 3 different questions that looks a little closer to what you were asking for, which maybe might get closer to the target when you throw the dart, throw the 3 different questions at it, you might get closer to the one that you're looking for. And it will return the context of all the 3 questions that you, your question and then the other 3 questions that you generated. It's going to return all the context and it's going to create a union of all those contexts and then be able to generate a final context that then gets passed to the LLM plus your question. So that way it gives you a more accurate answer based on the context that you saved into the vector database in this case. And so once that's done, and that's just a prompt on instructing the LLM how to do that here. And for us here, I have set that as a, 3 questions. I wanted to return 3 questions. And the reason for this is really to help the user overcome some of the limitations of the distance-based similarity search in this case. So after that, what we do is create a variable here called retriever from LLM. And essentially we're using that multi-query retriever, which creates multiple queries. And then from LLM, we are passing in the vector database as the retriever, which we have, where we've saved all our vectors. And then we have the LLM, which will be Mistral, which we've already picked there. And then the prompt will be the query prompt here that we just kind of created there to instruct it to generate the 3 questions. And then parser key, which is kind of telling it to every line after every line of question, create a, add a line break in between them. And then here we create the template, which is our prompt again that kind of tells the LLM answer questions based only in the following context. And our context will be what will be returned by those 3 questions, a union of them actually. And then the question will be the question that we are asking it. And I added some extra information here to tell it to add snippets of the context that they used to generate the final answer. So it's more of like citing itself in this case. It's not as accurate from my experimentation, but sometimes it gets it right. So worth a try. I haven't fully experimented into looking into how to solve that. Completely, but interesting maybe for some other videos in the future as well. And here the prompt itself—we're just compiling that prompt and adding this into our prompt template. And then here we're creating the LangChain expression language here LCL using chains. And then here the context will be the context from the retriever LLM tool, which you've kind of added up there. The question from RANNEVO pass-through, and then our prompt LLM, and then the string output parser will be the one returning the output. You can think of this as a chain; it's taking all this stuff, passing to the LLM finally from. The prompt and the question, and then finally returning the output here. And so the response here, what we just do is we invoke this chain itself that we've created. This is a chain itself. We invoke it using the invoke function, and then we pass in our question and it's gonna run through it and give us a response. And then we log in here to tell, to say question processed and response generated. Cuz after this response variable, it should already generate a question, an answer there. And then we just return the response from the function And then we get to the next stage here where we have the extract all page as images, and we're caching this data as well. That way we don't have to rerun it all the time. We you know the pages refresh for this case. So we're using PDF Plumber here, which we installed earlier. And so of course we log in at first, but then we're creating a variable here with an empty list, and we're iterating through the entire PDF, converting it into image, each page into an image and then appending it to this PDF pages and then returning that. And then the next stage here is deleting the vector database. And here is where this button right here on the app is created. So the functionality, at least for that app. So we are, you know, if the vector database is not None, which it means it's not empty, it has something in it. We want to delete all of the collections that are within it and also delete all the session state stuff in session state as well. Or at least pop the first one and delete them. And so that portion is done. We now go to the main part of the function, which is really implementing the Streamlit app. So far that was kind of like the orchestration layer. And maybe before I forget as well, I think there was one thing here that I maybe missed, which is the cache resource. So when we are extracting the models from our local system, which is really returning all the names of the models that we have on our system, we'll be using the cache resource. And in Streamlit, it allows you to cache resources like models and things like that. So here we're caching the name of the models that are already in our system here. So it doesn't redo that all the time and kind of take some time to do that. So let's go back to our main function down here. And this is where we kind of put everything together. So st.subheader, it creates this header that I was talking about earlier. And we're just writing st.subheader. We put in the emoji there. It's going to render the emoji itself. And then the Ollama PDF RAG playground. And then the divider here, gray, creates this gray line underneath. And you can change that to rainbow or whatever other color that you want to change really in this case. And the anchor here is false because I don't want an anchor because the page is— the app is just this long. There's no other many sections within the app. So you don't need the anchor to be able to kind of go to a specific section of the app here. And then the models.info variable here, it just kind of stores all the models that are model information that are within your local system that you've downloaded through Ollama. And then the available models, we're using the extract model names to be able to clean up that data for the model information and return just the names of the models themselves. And once that's returned, we create the 2 columns now of the app itself. So we have column 1, column 2, and the first column is bigger— well, the second column is bigger than column 2. So as you can see, 1.5, and the second column is 2. And that's kind of like the width really of the column itself. So this is 1.5, this is 2. So this is bigger than this first one. And you can change that however you want as well. If you don't want to have these dimensions, you can just delete this piece right here and just write 2 because you have 2 columns, just write 2 in there and it's going to split it in half equally. But in this case, I want the chat to be a little bigger because that's the most important part really of this app. And so here we instantiate some session state. Variables for messages. And then we have for vector database, we create an empty so that way we store sessions. So when the app refreshes, you're able to see that the model is still in there, that the PDF is still in there and the chat is still there even after the page refreshed several times since I've been kind of playing around with it. And that's the goal for that. And then if there's an available model, I want the user to be able to pick it from this down select box right here. And this is what this section does really. It's using a select box. To be able to display the models you have locally, and then file uploader, which will be in column one, is right here, and it just creates this section right here. And we don't want it to accept multiple PDF files; just one file is good. And then the type of the file we want it to be PDF, not Excel, not any other file. And then if the file upload or if the file has been uploaded, we just want to pass that the session state to say session state file uploader. Here's the file. uploaded. And then vector database as well. We wanted to create the vector database and add that to session state as well. And then extract the images and save that as well into session state. So we don't have to reload it all the time. And then we create a zoom level here, which allows us to zoom into the app itself, to the PDF itself. Sorry. And so zoom level, we are creating a minimum of 100 and maximum of 1,000. And the actual value is 700 when it loads the first time. And then the step is 50, just meaning every time you move it around, it just moves 50 up or 50 back. And then with the first column, again, we're creating an instance there to just kind of display this PDF. So we're using SDImage and then iterating through all the pages that we converted into images and displaying them within the application. And then for the width, we're just passing in the zoom level variable here that would allow us to be able to zoom in and out. There, and then the delete collection. We're creating a variable there that holds the button to be able to delete the collection, and this is it. And then the logic here to delete the vector database here as well, calling the function that we had created earlier. And then we have column two here where we kind of create the container message to be able to chat with the model, the the PDF itself. So here is the container that we create in that case. And we have these emojis and you can change. So you can see here, this cool one with the glasses, that's the human. And then the robot of course is the model itself. And we are just instantiating here, enter a prompt. And this is what it looks like here before you enter a prompt. It tells you kind of like ghostly, enter a prompt. And so that's what it does right there. And we're using that $rolldraw sign to be able to use the variable and also append directly there what needs to be put in there as well. And then we're saving that session state as well once a user enters the question in there or the prompt. And then after all that is run, after the process is finished, we display the output from the LLM here using Markdown, st.markdown, and displayed it just the way you see it on this app right here. This Markdown just displays that on the app itself. And then there's just a bunch of else statements here to catch some of the stuff that if If the ifs didn't work, then what happens after that? So please upload a file. That means you didn't upload a file already. And then session state here, saving the session state. We have the exceptions here in case something stops running and it does throw an error. It'll tell you there what the error is about. And that's it really. And that's just the full code there to generate this application, which is really cool to be able to see it in action. And the other big thing that I wanted to mention maybe before we close this out was to show you how logging looks like in the application itself. So if you can see here, the logging portion of our app when we ran it, you could see the logs here. So here we, first of all, the first question was, what is this document about? And so it's, it was using the Mistral model. And so it generated 3 different questions based off what is this document about? And the first one was, could you describe the main topic or the content of this document? And then the next one was what, Are the key themes or subjects discussed in this document? And the last one was, can you identify the primary focus or subject matter of this document? So you can see those are very related, but just slightly varied with different wordings that could really affect the output that you get from the LLM itself based on the vector database as well. And so once that's done, it went through the process there and then it finally gave us the information that the question processed and was answered. And then that was done. And then it just went into like the 200 section. And then the next question was, why is it important to have session-aware AI agents with memory? And also that was using Mistral. And then the generated queries here based off of that question was, what are the benefits of utilizing session-aware AI agents with memory in applications? And then how does having memory in session-aware AI agents enhance their performance and effectiveness? The last one is, in what ways does implementing session-aware and memory improve the functionality of AI agents? And that was done. And then it did process that and gave me the answer. So that's a gist of how that looks like. So this code, I'll save it in our GitHub repo and I'll be able to share it in the description below. If you have any questions about any piece or any portion of this application that we just built today, please feel free to let me know. And I'll be happy to answer any questions that you have and play around with it. Let me know if you like it or not, and yeah, provide me some feedback. Otherwise, see you in the next video, and goodbye.