Working with MULTIPLE PDF Files in LangChain: ChatGPT for your Data

If you are searching for a way to load multiple PDF files in LangChain to do information retrieval, then this video is for you. I'm going to show you how to load multiple PDF files into LangChain and use OpenAI's models for efficient information retrieval. So if you want to save time and streamline your research processes, keep watching. We're going to be using Google Colab. I will put a link to this notebook because before using this notebook, just make sure that you go to file and make a copy in your own drive. Let me walk you through the notebook. So here we are installing different packages that are needed. So LangChain Constructor, that's for reading PDF files. OpenAI, Chroma, DB S faster speed and Trick Token. Next we are importing two different functions. So one is this unstructured PDF loader that would be used for loading files. And then we are also loading Vector Store Index Creator. Now this is where all the magic is going to be happening. So I'm going to go into the details later in the video. Next, we need an API key from OpenAI so you can actually go to this link. You will need to make a account which is free and usually you get some free credits. So go to your account, click on API keys and then you can create a new one. Simply name your API key and click Create. And we're simply storing this in the environment variable. I'm going to delete this, so you're not going to see my API key. Next, using this code, we connect our Google Drive. So if you run this code for the first time, it will ask you permission to connect to your Google Drive. So in this case it's mounted, so I don't really have to do anything again. Now keep in mind you can run this whole code locally as well. Like you don't have to run it on your Google Drive. So what I did was within my Google Drive, I created a folder called Data and put two different PDF files in there. So here's my folder and then I took the State of the Union and simply divided into two different files. So I have part one and part two. Now, later on we are going to look at an example of Research Papers which has images as well, and see whether this approach can work or. So this code is simply taking the root directory as the Google Drive, then appending my data folder into the path and you can see there are two different files in here. Next we are using List Comprehension to load both these PDF files using two different data loaders. So essentially what is Happening is here we have a for loop and it's simply iterating on the number of files that we have in that directory, picking each one of them and creating a separate loader for them. So here you can see that we have two different data loaders. Depending on the number of PDF files that you have in your directory, you will see the number of data loaders changes. Next we will look at Vector Store Index creator. This one line of code is doing a lot of heavy duty work. Now using this function, we are loading all the PDF files from these different loaders. And then there are three main steps that are happening after these documents are loaded. So first, documents are split into chunks. Then for each document this function creates a embeddings. Then these documents and embeddings are stored in a vector store. In this case, by default, the vector store is Chroma db. In my previous video I showed you how to do these steps manually. But in this case this single function is taking care of everything in the background. It's using OpenAI's text embeddings. We will look at that in a bit. So let's run this code. Now you will see this warning message. Because Dektron 2 is not installed, I was having issues with it. If this library was installed, it was giving me errors, so that's why I didn't install it. But that's fine. For the time being we will simply ignore these messages. Okay. Now in order to retrieve information from our vector store, all you have to do is you use index query and then whatever query or prompt you have. Right. So in this case what is the main topic of the address? The main topic of the address was investing in America and rebuilding infrastructure. So we would look, we are looking at State of the Union address and let's look at a little more complex example. In this case I have another folder called Data underscore two. And here we have two different research papers. So this is from the lip sync wave to the paper and then GPT for all. And these are the type of documents you probably are going to be working with. So there are images in there and a whole bunch of text. Let's do information retrieval on those. So the main steps are exactly the same. First we are simply defining the path and we see there are these two documents. Then we are creating our indices and creating a vector store for this. And here I am actually asking it a simple query. So index query. How was the GPT for all model trained? Right. So I get a response. GPT trained model was trained For Laura. And here are the number of parameters. Now let's say if I have multiple documents and I want to see which document contains this information, right? So I actually can use this query with source, with sources, right? And if I now give it the same prompt, right? So apart from the response that we saw, it will also tell us the source. So if you remember, there are two papers, but it got the actual paper where this information is available. And that's pretty neat because if you want to look at the sources, you can actually do that. Let's ask a question regarding this lip sync paper and I'm asking it who wrote that paper? So it returns the lip sync paper was written by and here's the list of authors and then added the source as well. So that's the file name that we have it in here. So it's pretty neat. You can easily work with documents that have images in here. Now let's look at what is actually happening under the hood in this vector store index creator. And it is important to understand because in case if you want to change any hyper parameters that you're using or you want to use different model. All right, so we will be using a LangChain documentation for this purposes. There are three main steps. So first you load a document, then you split the documents into different chunks, right? So in this case it's thousand characters chunk and you can define the overlap or you want to not overlap, it's up to you. And then you simply split that. So you're going to have multiple documents in here. Then you want to select your embeddings, right? So in this case they're using OpenAI's embeddings, but you can replace it with any other type of embedding that you want. Now, using these embeddings and the text documents that you created with different chunks you want to create in the vector store. So by default Chroma is used for the vector store, but you can replace that with some other vector stores. Now next we want to retrieve information from our embedding and vector store. So for that you simply create a retriever object, right? That is the retrieval interface that we will be using next. Putting everything together. So we simply create a chain which has a large language model. In this case OpenAI, I think it's a text DaVinci and then the retriever interface that we provided. And next you simply run the query to get a response. And the vector store index creator is a simple wrapper around all these steps. So you can actually pass these different parameters so for example, in this case we're passing is the vector store, where is chroma, the type of embedding and then the type of chunk creator or character splitter that we want to use at the end I went into a little bit more details than I usually do, but this is important so that you are able to understand it and modify these things based on your own needs. I hope this video is helpful. If you have any questions, please comment below and I would try my best to answer them. Consider liking and subscribing to the channel. Thanks for watching. See you in the next one. Thank you.