Step-By-Step: Add 100+ Files to Pinecone for RAG AI Agent with n8n
Dr. Violet Howard shows how to upload 100+ PDFs or Google Docs into Pinecone using n8n, then build a RAG chatbot that answers questions about the data. The workflow splits files into 1,000-character chunks, generates embeddings with OpenAI, and stores vectors in a Pinecone index under a "restaurant" namespace. The final agent uses GPT-4.0 and memory to answer staff questions about menus, reviews, and policies.
Chapters
A few weeks ago, I made a video about an RAG AI agent where we pushed a PDF into a vector database, and then we were able to chat with the agent to get answers from the PDF. So after that video, a lot of people started asking me, well, what if we have hundreds of PDFs or hundreds of different documents that we want to push into a vector database and have that database keep growing? And we don't want to have to manually, you know, test workflow every time in order to send it through into Pinecone every single time. So in this video, that's what we're going to be doing. And it may be a lot simpler than you think. If you haven't already watched that RAG video, I would maybe do that real quick so that you understand the whole process of retrieval augmented generation and pushing data into Pinecone. You'll understand indexes and namespaces a little bit more, and then you can come over here and then this video will make a bit more sense. And also like setting up Pinecone and the API keys and all that kind of stuff. And finally, before we get into the video, just wanted to say thank you guys so much. The amount of support and feedback I've been getting from you all has been overwhelming, and I've been super happy to hear that you guys are learning from my content and enjoying the content. Um, please hop into the school community. The link will be down in the description. It's 100% free. It's really nice to get to interact with you guys, and I'll see you in there. So for the purpose of this video, I'm going to be pretending that I own a restaurant and I have all this information that I want to use for an internal sort of agent where my staff can ask questions about our current reviews, current menu changes, promotions, uh, policies, stuff like that. It can also be great for training. But what we have here is all the information. I have a bunch of Google Docs that I had ChatGPT write up some dummy data for me. And so we've got these documents, and then I just wanted to put them within a single folder in my Google Drive. So I've got a folder here just called Data, and I've got all of the documents in here. And this is really how we're going to be able to get everything we want into Pinecone without having to manually do it for each you know, each document, each file, whatever it may be. So now that you've got all your information that you want in your database into a single folder, let's hop into n8n. Now that we're ready to start building the workflow, we have to add the first step, which is always gonna be a trigger. So we're gonna do trigger manually here just so we can hit it, test workflow, and then it will, you know, search the folder, grab the files, put them into Pinecone for us. But in the future, once you already have your original database set up and all you need to do is add more information to it, you could have this trigger be like a Gmail folder trigger. That way, every time you add a file to that folder that it's pulling from, then it will run through and just put it in the database for you automatically, so you don't have to come back here and test it manually. But that's, um, you know, in the future. Right now we're just setting up the original database. So we've got our trigger. Now we want to go and grab a Google Drive node to be able to actually find the folder that we're searching through. So we're going to grab Search Files and Folders. I know it's a little confusing because you see like Download file, you see these folder actions, but we're going to be searching files and folders to find the folder, and then we can have another one after this to download the information. So as always, configure this node, get set up with your account, the resources file folder. The operation is that we're searching for one, and then you have this query here, which is a little confusing. Just don't touch it. This is where you may think you need to put in parameters to find a folder, but all we're going to do is we're going to go to filter and click on Folder. Then we have From List, which is super nice that NNN lets you just choose from a list. And the only folder I've got in my drive right now is Data, so we're going to grab that one. I'm going to hit Return All because we want to get all the information back from this folder. And then we'll just hit Test Step and we'll see— I think it's 14 documents. So we've got all 14 documents coming back here. We'll switch to JSON real quick. As you can see, all we're getting back about these folders is we're getting the ID, and then we're getting the name of the folder. So we got feedback, special events, reservations, loyalty program, promotions, stuff like that. But it's just important to know we're not actually getting any content from these files here. All we're getting is ID and name. So that's why we need to come in here and grab another node, which is going to be another Google Drive. But this time we're going to grab the download file because this one will actually give us the information from the file. So I'm just going to call this one Get Content just so we know what's going on. We'll have this configured already. We're looking for a file, we want to download the file, and instead of choosing from a list, we are going to do by ID. This way we can just grab the IDs that are coming in from the previous node. So we'll hit schema up here in the top left, that way we can drag and drop the ID right here that was from the Google Drive node previously. And now we have ID, and it's going to be an expression, so it'll get content for every single ID of a file that's coming in from the previous Google Drive node. So we'll hit test up here. This one's going to take a little longer because it's going to be searching through those 14 files and pulling the information back. So we'll just give this one a second. Okay, this just finished running. As you can see, we've got 14 files. All of them are here. This last one is for orders. The 15th one, or 13th one is for staff. This one's for reservation policy, as you can see. But what's important to note here is that this is coming out as binary. So JSON, we're just getting the original IDs and names, but the binary data that we're getting is the actual content. So that's important to know, um, for future steps. So just keep that in mind. This is binary data. So now what we want to do is we want to add a loop. And we're doing it this way because, you know, I have 14 files, but if you did have hundreds and hundreds, I don't know if you'd be able to just take all 14 and put them straight into Pinecone. I haven't tested with hundreds, but you definitely could do that. I've done that in the past where you don't loop and it's fine. But for the purpose of this video, let me just show you how the loop is going to work. It's going to be taking batch sizes of 1, which basically means it's going to grab the first file right here and then it's going to run it through Pinecone once we configure that. And then once it's, once it's put that first file into Pinecone, it's going to come back here and then it's going to grab the second file and put it in Pinecone, then grab the third one, put it in Pinecone. So that's how it's going to work and we'll see all of that play out. So we've got the loop set up here. We'll get rid of this and we're going to set up the actual loop. So in here, what we want to loop is the Pinecone vector store. We're going to add a document, set up your API or sorry, your credential, which is an API key. In Pinecone, obviously you've got your indexes and then you have your API keys right here. You'll just copy this value and paste it back in n8n. That's all you're going to do there. The operation, as you can see, we can retrieve, update, or insert. Right now we're just updating our database or configuring it, setting it up, so we're going to insert. And then in Pinecone, I don't know if you saw, but my index was called sample, so we're going to grab sample. And then finally you have the option to add this to a namespace if you want. I think it's beneficial to keep all your information organized. It also can help your agent find things quicker if it knows which namespace to look within each index. So we're going to be calling this namespace restaurant because it's information pertaining to the restaurant. And now we're good to configure the rest of this node and then we can test it out. So we're going to grab an embedding, set up the credential. When we set up our index called sample, we set it up with the embedding of 3 small. So we're going to choose 3 small. And then this is where I was talking about, we need to keep in mind that the information we're getting from each file is binary. So we're setting up our document loader. We've got default document loader right here. Types of data coming in. We want to change this from JSON to binary because the JSON, all that's processing, if you remember, was the ID of the file and the name of the file. But the binary data, that's going to be the actual content that it's going to find within each document. So we've got binary, we can leave that as is. And then finally, all we need to do is add a text splitter down here. Character text splitter. We're going to leave the chunk size— or sorry, actually, I don't want to use character, I want to use recursive because this one sort of keeps things like, as you can see, split text into chunks by characters recursively, recommended for most use cases. It's going to keep the context of what's going on. So if you— if it's getting chunked in between sentences, it'll kind of keep those related. So just better for giving the agent context of what's going on rather than just a ton of information with like a token splitter. So we're going to do recursive here. We'll just keep the chunk size. It's going to grab 1,000 characters at a time and it's not going to overlap anything. So we're going to keep that as is. We'll hit save. And then finally, we need to make it loop back. So after it grabs the first item, goes through here, and then it would just end. But we need to grab this and put it back to the loop so that it will keep going until all 14 are done. So let's test this out real quick. As you can see, it's going to search the folder. It got all the IDs. Now it's going to grab the content. This is again going to take maybe 30, 40 seconds. So we'll give it a sec, but then we'll see it actually loop through. Okay, just finished. Now it's got the first item, it's embedding it. Second item, third, fourth, fifth. So as you can see, it's going through and doing that for each one. So this is the way you want to do it. If you have, you know, 100 documents within this folder, it'll grab all 100 here and then it's going to loop through and embed each one into your vector database. So that's done. Let's head over to Pinecone. This is our index called sample. We'll click into it and we can see the data that's come through. We've got our namespace called restaurant and it's got 14 vectors in there. So we're good to go. Now we can hop back into N8N and then just quickly set up a super simple agent in order to talk to our restaurant information and see what's coming back. So new workflow here. This is gonna be a demo restaurant agent. Once again, we need to set up the first step, which is a trigger, and we're gonna be talking to this agent. So we're gonna set up a chat message received. We're gonna come in here and add the actual agent itself. So AI agent, and let's just keep this one as a tools agent for now. We can do a simple prompt once we get things set up. The chat model I'm gonna be using for this agent, I'm pretty loyal to the OpenAI, I've been using the 4.0 pretty much for everything unless maybe it's a smaller task, like labeling things, but 4.0 for reasoning and for conversational things. So 4.0 here. Memory, I'm going to grab the window buffer memory. Super easy to set up. You literally just click on it and it's going to give your agent context of the conversation. So the context window length is going to be 5 chats is how much it will remember. If you don't set this up, it'll It'll still work, but you won't be able to reference a previous question. It's just going to reset its memory completely after each question-answer chain. So we've got our memory, we've got the model. Now we just need to add the tool, which in this case is a vector store. So we've got our vector store. We'll just call this data. The description for this tool is going to be call this tool to, let's say, access the database to answer the user's question. Okay, so that's good to go. We need to add the model real quick. Once again, OpenAI. We're going to grab R40. And then we need to set up the vector store. So we did Pinecone. It's right here. Set up the credentials. This time we're not inserting. Obviously, we're going to be retrieving for agent. So we've got that, and now we finally just need to set up the embedding. Once again, we did 3 small, so we're going to come in here and just grab 3 small real quick, and we should be good to go. What else did we forget here? Oh, the actual index, of course. So the index is sample, and we need to make sure it's pulling from the namespace of restaurant, and make sure that that's spelled the same way it is. So restaurant, restaurant. Okay. Restaurant's a tough word to spell. I always have to Google it, so we're good here. All right, so I'm just gonna give this guy a super quick prompt and I'll be back in a sec. So I just came in here and said, you're a restaurant assistant. Your job is to answer questions from the staff about the menu, feedback, policies, hours of operation, et cetera. All of this information can be found in the Vector Store tool, so call that tool each time you are asked a question to ensure you are providing the staff with accurate information. Please be friendly and throw in some jokes and emojis. So this is not the optimal way to prompt an agent, obviously. For the sake of this video, you know, I just wanted to show you guys how to get a ton of documents into Pinecone in one fell swoop rather than doing it all manually. So this should suffice in this case, but let's talk to this agent and see what it's got for us. So what's on the menu? Let's just try that. I know it's really simple. Let's see what it says. Okay, so here's the delicious lineup we've got for you today. We've got appetizers. It even gives us the price. We've got main courses. Tiramisu. Heads up cheesecake lovers, cheesecake is not available today. Feast your eyes and your taste buds. So it gives us information. It's friendly. It threw in some emojis. Let's see what else we can ask it. Let me just go back to these documents. We've got special events, loyalty programs. Let's ask about some customer reviews. So what are the customers thinking about? Us. Okay, so here's what our fabulous customers are saying about us. John Doe was impressed with Michael Brown, our waiter, praising him for being attentive and polite. Jane Smith gave a shout out to Sarah Williams, our chef, for preparing the steak exactly as she wanted. And then at the end it says, looks like we're serving satisfaction with every dish and service. Keep up the great work, team. Nice. Let's just do one more real quick about, um, I don't know, let's, let's ask about the suppliers. So who, who are our suppliers? Okay, so yeah, I know this is a simple example, simple data, but just goes to show how quick it is to get something like this set up. And once you have something like this set up, how you can expand on it. But let's see, we got our culinary creators are supported by some top-notch suppliers. We've got fresh farm produce, Laura Green. We've got prime meats from John Carter, Baker's Best Lots of emojis, very colorful. With this dream team, it's no wonder our dishes are always a hit. Okay, so as you can see, it's doing what we wanted. It's friendly, giving emojis, and that was just a really simple prompt, but it's working as it should. It's grabbing all that information from the vector store tool, which we set up really quickly. We put 14 documents in there with a loop. So I hope that that answered your guys' questions about how can you get, you know, lots of documents rather than just one into, um, Pinecone, the things to keep in mind: the first time when you're searching through the folder, just remember it's not actually grabbing content, so you need to do another node to grab content. And then just remember with, um, the content coming through, just making sure you know if it's binary or if it's going to be stored in JSON. That way you're embedding it correctly and you're actually getting information you want. So with that, all I've got for you guys today. Really appreciate all the support. Like I said, again, I'm excited to keep making videos for you all, so let me know what you want to see. And thanks, guys.
