How to chat with your PDFs using local Large Language Models [Ollama RAG]
In this tutorial, the presenter demonstrates how to construct a completely offline Retrieval‑Augmented Generation (RAG) system that lets users chat with the contents of PDF documents using locally hosted Large Language Models (LLMs) via Ollama. The workflow begins with loading PDFs through LangChain’s Unstructured PDF loader, then recursively splitting the text into overlapping chunks to preserve context. These chunks are embedded with the Nomic Embed Text model (served by Ollama) and stored in a ChromaDB vector store. A multi‑query retriever generates several paraphrased versions of a user’s question, retrieves relevant chunks for each, and merges the results into a concise context. This context is fed to a local LLM (e.g., Mistral, Llama 2, or Gemma) via LangChain’s ChatOllama wrapper to produce the final answer. The video also covers installation steps for Ollama, model selection, vector‑store alternatives, chunk‑size/overlap tuning, and the rationale for keeping data offline for privacy‑sensitive documents. Finally, the presenter outlines plans to wrap the pipeline in a Streamlit UI and mentions future extensions such as agent‑based RAG strategies.
Chapters
In this video, I'll show you how to build a local RAG system that doesn't connect to the internet using Ollama and Python. So there's so many reasons why you want to build a local RAG system over using something that's connected to the internet. So maybe you have very sensitive documents like your medical documents or your financial documents. So security is very important there. You don't want to share those very detailed personal stuff on the internet because you never know which LLM is going to be trained on. So the best way for you to do is be able to get models that you can use offline using Ollama in this case. And I'll demonstrate how to do that, how you can feed those personal documents to a local model that's not connected to the internet and still be able to chat with those documents, especially PDFs in this case, and make it very easy without having to upload it to a public online system that will be able to use your information for training. So let's get started. All right, so let's get to it. So this project right here, as you can see on my left side here, I have PDF files, and this could be files on your system that you've saved, maybe your resume, maybe some lecture notes, some books or whatever that you want to chat over using the LLM. And in this case, these are documents that the LLM wasn't trained with. So this is data that LLM has never seen potentially, and you want to be able to chat with that. And the benefit of this, like I mentioned before, was These are private documents that maybe you don't want to share with, you know, using public LLMs, things like ChatGPT, because they might, you know, they'll have that data. They have an option to have that data and be able to use it. And you don't want people maybe to see some of your sensitive data. And so all this will be happening locally in your own system. And this is the entire pipeline, as you can see here. So we'll start from here. You load your PDFs using unstructured PDF loader, and this is LangChain. And most of the functions we'll use here and the modules will be really, LangChain modules. So LangChain, if you're not familiar with LangChain, is an AI Python framework that helps you build AI apps by— they abstract a lot of the work that needs to be done in terms of like loading files, using LLMs, and so many other things combined as well, like chunking and things like that. So they make it really easy by abstracting that away. So you just focus on building the app and makes it easier as well. So in this case here, we are loading these files using LangChain unstructured PDF loader. And Unstructured is actually a company, unstructured.io, and their product is they will let you load any type of file to be used by an LLM. And that's really cool. So here LangChain wraps that and use that. We use that once we load the file, it's able to extract all the content from the PDF files. And then what we'll do after that, we'll also use another LangChain function here that will go ahead and recursively split all the characters. So basically it will do chunking for us and we'll have a strategy there of how many characters we wanted to chunk and also overlaps as well. And then once the chunks have been chunked, we'll put them in a variable and then we'll iterate through them and then we'll get the— we'll use Nomic embed text. And this is an embedding model and there's so many other embedding models that you can use, but for this case I chose Nomic because it's just It's just fast and it's one of the popular ones. And just as well as a note, everything in this pipeline right here can be replaced. So you can switch this with any other ones that you prefer. So this is not like a, you know, this is the best kind of thing, but no, this is just to get you started and know which ones you can, you can swap. So you can swap a bunch of this stuff. There's so many options. And then after we've embedded, we've created vector embeddings of those chunks, we then load them to ChromaDB. And as well, again, You can use Chroma, you can use Weaviate, you can use Milvus. There's so many of them out there, but I'm using Chroma in this case. Now once you load them to Chroma, the next step would be to query it and retrieve whatever you want to ask of that data, right? So the user, as a user, you'll put in a query. You'll say, hey, maybe, you know, what is this document about? Or what, what's, you know, summarize this for me. So what happens here in this case, I'm also using a multi-query retriever module from LangChain. And basically what that does is optimize your question more. So basically you might ask, summarize this for me. And this multi-query retriever, what it does, I told it, hey, generate 5 more questions based off this user question. And it will go ahead and generate questions that might be varied differently, worded differently, stated differently. And that way when it sends those 5 questions to the vector database to be able to retrieve the close enough context that matches those questions, it returns also like another 5 answers for that as well. So that way, once those questions come back, we get a union of all the questions and get a short summary of like probably the best answers of all the 5. And that's what gets sent to the LLM for it to generate an answer for you. And that's just a strategy to make your answers better in this case for RAG. And there's so many strategies for this piece for RAG actually. There's so many strategies. There's things like agents, which is something that is recently gaining popularity, but I'm using this multi-query retriever here to make it a little better and see if it does better than just kind of regular, just retrieving it without doing this. And so once those questions have been generated, then it gets passed to the LLM with the prompts, which are questions, and then the context will be those summaries. And then the Gemma here is a model that I'm using locally, but in this case, I swapped it with Llama 2. And you can swap it with whatever local model that you have here from Ollama. And then once it does that, it gives you back the response to you right here as a person on the other end. So let's see how the code looks like, right? So let's jump over to the code. And so what I've done here already, I have 3 different sections here that we'll go through. So the first one is ingesting the PDF. The code for ingesting the PDF. And then I have the code for creating vector embeddings of that PDF and then retrieving, which is asking questions over it and getting answers back to you as well. So let's start with ingesting PDFs. So what do we want to do here? First, we want to install unstructured and LangChain and also install unstructured all docs. And this all docs basically covers everything to include PDF and text as well. And then The next thing here we want to import, we want to import unstructured PDF. So here we installed it, but now we want to import it so we can use it within our code here. And then we also import online PDF loader. And what this online PDF loader does, it gives you an option. Maybe, you know, if you don't want to, if you don't have these files locally, you can access it by downloading it from the internet where there's already a PDF hosted there. One thing to remember though, if you don't have access to that PDF online, like maybe you might click it on your website and you could order on a website and you can see it, But when it comes to downloading it, it doesn't let you do that. It will throw like a 403 error, which basically means you're not allowed to access it. So keep that in mind. So it might just be best to just download and then use it. And there's also lots of different loaders within LangChain that you can use that might do that better than this one. So I downloaded this file from online. I think it's from World Economic Forum. And it was a recent one from January. And I figured, hey, these LLMs were trained a while back. And so they won't have this information. So what's the best way to demonstrate how you can add context to LLMs by using RAG? So this is the best way to do it. Give them something that they haven't seen, they haven't been trained on. So this is the file that we'll be going through. And I think it's about how many pages? It's about 26 pages. So not too big, right? Just good for beginning understanding RAG, how it works and implementing it. So cool. So the next step here, what we will do next is load this file. So it's pretty easy. So if local path, and basically I'm just saying if this local path is, you know, has something, you know, with a path to it, I want it to load and I'm using a structured loader and it's loading that. I'm passing that as a loader and it's loading that. And then once that's loaded, let's just verify here. This is just verifying the first page and we've just previewing, hey, what does that data look like? So we can see this is already loaded. So it definitely just took, went ahead and we can verify that as well. So it says in collaboration with McKinsey and Company, the Global Corporate Barometer 2024. So let's see, and that should be page 1. Let's see if that's exactly what's on there. All right. So yeah, you can see in collaboration with McKinsey, the Global Corporation Barometer 2024. Okay. So we, it has the content that we're looking for. So cool. All right. So we're done with ingesting and we verified that, hey, We did ingest. We can see some data, which is cool. All right. So now let's go to the next step, which is vector embeddings. So we want to embed this text and basically we're converting it from human readable to computer readable. It'd be like zeros and ones and more like vectors, right? So these are something I want to note. So for vector embeddings, you need to also use a vector embedding model. And in this case, I chose to use Nomic Embed. And if you have Ollama, If you don't have Ollama on your system, the quickest way to do it is just go to ollama.ai. And then from there, and we can go here right now, I can show you how it looks like. So you can go to ollama.ai and you can install this. You just download it here. You click download and you download in your system and install it. And if you also have Windows and Linux, I think it's on preview right now. And it's, it's really easy to install. They have like some other instructions on how to do that right now. And once you've installed it, the next thing would be for you to pull models or install models within your system. So there's a list here. So if you come up here to models, which is ollama.com/library, you will see all the files, all the models that are available for you to use locally. So we have Gemma, Llama 2, Mistral, Mixtral, Command-R. You have LLaVA. So many of them right here. So many of them here to play with. We're gonna be using Nomic Embed Text. And as you can see here, it's a high-performing open embedding model with a large token context window. Yes. So that's the key part. Has a large context window that we can use. And that's the key part right there. So you can take in a lot of tokens that we can then load it to our vector database. So here I've already installed this, so I don't need to install, but this is how you would do it in a vector— not a vector database, on a Colab notebook. And then once that's done, you do ollama list, and that way it'll show us all the models that are within your local system. So if you have, if you just installed Noamic up here, if you do ollama list, it'll show you if it's there already. And you can see here it's already in my system and I have some other ones as well that we'll use down the road. And after that, the next thing would be for you to install ChromaDB, which is a vector database that will store all the vector embeddings here. And then the next thing you wanna install is LangChain text splitters. And this is what we use to chunk our text. All right. So the next thing here we do is import Ollama embeddings and then also import recursive character splitter and then Chroma. So the first step here, we are doing split and chunking. So we'll split the text and the chunk size that we'll split here, which is chunking really here, we'll do 7,500. Chunk. So basically every character, 7, after 7,500 characters, it will cut there. And then that'll be the first one. And then it will do another 7,500, cut it. That's the next one. But there's always a problem, right? So if it does cut at 75, it might cut in the middle of the word or in between words that are actually making sense. And then by the time it starts the other one, it might not connect very well. When it comes time to retrieval, you might not get the accurate answers that you're looking for. So it might hallucinate or just not really get to understand the full context. So we're introducing here chunk overlap, basically to kind of take some text before, like in between each one of them will kind of have like a middle portion where we pass in between each so that they remember, oh, this was the other context from the other line and this was from the other one. So that way it's, there's an end and there's a coherence. So that way it's, we are just not kind of lost in the middle here, avoiding the problem of lost in the middle a little bit. So it's not, 100% foolproof. You can play with these numbers. These numbers are arbitrary. You can just adjust to find the best that kind of work for you as well. And another thing as well, you can fine-tune your embedding model and that's advanced at this point based on your data to make it better. So there's so many strategies here to use, but this is just a basic one to show you how this is done so you can get going and working with this. So the next thing would be to, once you've done that, it would be split the documents. And here I'm just passing the data that I wanted to split. And we're passing in the data that we already loaded at the top, which is our PDF. And it's going to do the splitting into chunks. And then the next thing is to add that, all those chunks into vector database. But first thing, what it does, it takes every chunk and then it converts those to vectors and then it loads them to the vector database. And so basically here, what we're doing is saying from documents Chroma, hey, Hey, Chroma, from my documents, please take in the chunks. That will be the document. And then embedding, we'll do Ollama embeddings and we'll pass in the model that we want to use as our embedding model, which is Nomic Embed Text, which we did download a while back ago. And then we also wanted to show the progress. So while it's loading the files, we wanted to show the progress when it's working through it. So that way we're just not seeing like empty screen for a long time, not knowing what's happening. And then We also want it to be in a collection called local-rag and local-rag. You can think of this collection name as more of like a database. So in your, in your database, you have like tables and that could be one of them. You know, you're putting a table, specific table. All right, cool. Now we're done with vector embedding. So just to preview a little bit what we've done, we've loaded our text up here. We've then gone ahead and created vector embeddings of that. Data that we just loaded from the PDFs. And then we loaded into ChromaDB for vector database. So the next step would be to retrieve, you know, you ask it a question now and then you get responses back. So there's a structure here of how we would do this with LangChain. And LangChain has chat prompt template and then prompt template. We'll use those here in a second. We have string out parser as well. And then we have chat Ollama, which will be the local model that we want to use. And then we have runnable pass-through, which will be our question. And then we have multi-layer. Multi-query retriever, which is what I talked about generating five questions, and that's what we'll get here into here in a second. So first, let's instantiate what model we want to use. So here, the local model that I'm going to use in this case to as on my LLM will be Mistral. You can change this as well. You can put it to Llama Two or whatever ones that you will download. Maybe your system has a better enough memory, or you have a bigger system, a GPU or something that can run some really good models here real fast. So we'll pass that to ChatOllama and we'll instantiate that as our model that we'll use here. So I'm going to be using Mistral. I found Mistral to be really good. So feel free to experiment with others as well. And then we'll create a variable here, query prompt, and we'll use a prompt template here. And basically this is what I was talking about. You know, you, the input variable will be a question. So it will take in your question and it will generate 5 similar questions based off your question. And from that, those are the questions that would be refined and will be used to ask, to pass those as queries to the vector database for the vector database to retrieve answers with those 5 questions. And at the end of that, we will get a union of all the answers you got from those 5 questions. And that will be the best probably answer to our question essentially. So that's what it does here. We just kind of passing in the question and then it's going to generate that. We're just giving you the instruction here and how to do that. And then after that, we're doing the retriever. So we're creating our retriever here and we're doing multi-query retriever from LLM. And we are passing it in with, we're going to the vector database, which we already created up here and already passing the text into that vector database from our PDFs. And so we are saying, hey, vector database, we are gonna use it as a retriever. And the LLM that we will use here will be Mistral and then the prompt query. This is the prompt query that we'll use. Hey, we want you to generate 5 questions that will be used. All right, cool. And then the next step would be to create a RAG prompt. And the RAG prompt here in this case is just saying, hey, answer the question based only on the following context. The context will be what we get from the vector database. And then the question will be our question that we asked in the beginning, right? The query. And after that, well, that will be our prompt. So as our prompt, To the LLM will be hey, will be from template. We just pass out as our template as well. So here we'll create a chain. So LangChain has these chains that you can use that makes it easy to chain things together. So we have the context will be our retriever, which we've set it up here already, which is multi query retriever that will get all those five questions and give us the context back. And then our question will pass our question to the runnable pass through. And then after that. We're passing the prompt and our prompt up here will be what we already made here, which is, hey, answer questions based only on the following context, right? And then we'll pass it to the LLM, which would be Mistral in our case. And then it will output the answers to us right here. So that's the chain right there. So we'll run that and then we'll get here and we'll just do chain invoke. And this is where we ask the question. So I was a little bit extra. I wanted to use input. And in Python, if you do input brackets, something in there, it basically gives you the option to, you know, opens up an input box, which starts blinking, and then you can type in a question in real time and then send it. You don't have to do that. You can just check out this input and use brackets. And I've shown down here how to do that as well. And so a quick note here, the reason why I'm already like this, this already has been ran before. I actually recorded this video in the first go-through and It actually competed with the system in terms of resources and it broke the video. The video disappeared. So wasn't able to save it, which sucks. Learned a lot there. But it consumes a lot of resources trying to run this. And if you have a GPU, the better for you, but I don't have that. So I had to run this first and then come record now. So, but essentially here, what I asked was, what is this about? Basically, what is this document about? That I just have in my vector database. And it says, hey, this document is the Insights Report of the Global Cooperation Barometer 2024 by the World Economic Forum in collaboration with McKinsey and Company. It provides an analysis of the state of global cooperation across 5 pillars: trade, capital, innovation, technology, climate, natural capital, health and wellness, and peace and security. The report examines trends in cooperative action and their outcomes to determine overall level of global cooperation in each area. It also includes recommendations for leaders on how to reimagine global cooperation in a new era. So that sounds pretty right. And it sounds like a summary as well. So this is another way of like, it's me asking for a summary. And I think it, from what we discussed a little bit earlier, it took what is this about and probably generated 5 different questions based off a variation of those and asked the same question. And that's why it sounds like a summary because I would think one of them was like, hey, generate a summary of this. And that's why we got this really nice summary. The next question I asked it was, what are the 5 pillars of global cooperation? And first of all, let's just go to those 5 global cooperation questions here, the answers that I expect. So the global cooperation parameter, 5 pillars of global cooperation. So the first one is trade and capital, and then we have innovation technology, and then we have climate and natural capital, health and wellness, and then peace and security. So let's go and see if it got that right. Right? So the 5 pillars of global security, global cooperation are one, trade and capital, innovation technology, climate and natural capital, health and wellness, and peace and security. Yeah, I think it did pretty good. It got it all right. All right. So this is just a simple example to show you, right? Like how you can do this really easily. And the most important thing is most of this stuff is swappable. So you can even use Llama Index. You can just not do it with LangChain and you can do it LlamaIndex or you can do it just raw. Don't use any of them and just do it by yourself. Right. Although you'll, you have to manage a lot of things. You have to build a lot of functions to do some specific things for you, like extracting PDFs and they might not even be consistent. So there's so many things in there to consider. And also there is the agentic things that are happening now that you have agents in between, maybe before you embed it. And also you have them between, within embedding, you might even assign an agent to each document. So that way you have one agent for each document that you can query. So there's a lot of, I guess, techniques and strategies being used right now for RAG that are coming up or being improved. And so this is just a basic example to get started. I might get into some of those, especially the agents ones, because I'm really interested in agents. Very recently I'm really interested in agents. So that might be something I might do in future and make a video of. Let me know if you like it, by the way. But for this one specifically, I just wanted to show you how you can get this done real quickly. And none of this is connected to the internet. You can do this offline. Literally, you can just, you know, so long as you've imported everything that you need to import up here, you can just log off, you know, turn off your Wi-Fi and you can run this basically on your system. So the next step that I will do after this, and I'm working on it, is converting these to a Streamlit app where it's very easy, user-friendly, because for some of the Doesn't code. This is probably very scary a little bit because like what's all this code? Like I don't understand. So I'll convert this to a Streamlit app where you can just basically you have an upload button, you upload a PDF file, and it does these embeddings in the back. It then it has a section where you put in your question, and then you just chat with that PDF basically. And this should be able you should be able to spin that at least locally on your system. This is not going to be on the internet as well, but it takes all this code that seems like very scary. And it gives you an interface where you can interface with it. I also want to note that Ollama has a web UI as well that you can use. But the reason why I tried using Streamlit is because it helps you, you know, you could just customize it the way you want it to. Maybe you don't like the way the web UI is built with Ollama. You want to build your own thing, or, you know, it's fun anyways, right? To learn and build something new. But That's all I wanted to share for this today. Please let me know if this is something that you enjoyed, and also let me know as well what you would like for me to see— what you'd like to see me cover in the next videos. And with that, happy coding and see you in the next video. Bye.
More from Tony
Summarize Videos and Research Companies with CrewAI + Gemini Models [Quickstart Guide]
How to Make Your AI Agents More Reliable with CrewAI Task Guardrails (Step-by-Step Tutorial)