Vector Database Optimization with n8n: Metadata, Text Splitting, & Embeddings

When building out AI agents, the number one most important foundational aspect is having a good knowledge base. And if your knowledge base is a vector database, you really need to understand how you can get your information into that knowledge base optimally so that your agents are able to search through it and give you accurate answers through RAG or other functions. In this video, we'll be using N8N and Pinecone to go over essential concepts like metadata, embeddings, and text splitting. If you guys find this one helpful, please consider giving it a like. It really helps me out. But let's not waste any time and get straight into the video. Tons of examples to go through. As you can see, we've got all these workflows set up and each one is going to do something different, so hopefully they'll all make sense by the end. But first things first, if you haven't set up Pinecone, you're going to want to do that. So you'll hop into Pinecone, um, pinecone.io. You'll create an index, you will set up your embedding. So just really easy to set up. I would just go with serverless and I would go with embedding 3 small, and then you'll just grab an API key and hook up to Pinecone. You'll be good to go. It's super easy, super cheap. And fast to set up. Then we just need a document to test with. So in this case, we're using this one. It's really short, really simple, because I want to be able to really explain what's going on with our splitting and our embedding and all that kind of stuff. So we've got document ID 1234 and then just some fake words. So anyways, now that we have that set up, we're good to go. So like I said, basically the principles are you're going to download some sort of file and then it ends. So we're downloading that file right here. And then you're going to put it into Pinecone. And when you're putting it into Pinecone, you need to make sure that your embedding is correct. So right here we have text embedding 3 small because in Pinecone we set up our embedding as text embedding 3 small. You have to make sure that matches. I'll show an example of what happens if it doesn't match. And then from there, you're going to make sure you're loading in your data. You can load it in using JSON or binary. Again, we'll show examples of all this. And then finally, you need to split up that text or split it up, whatever it is, you need to split it up somehow. And right here we're showing recursive character text splitting. As you can see, there are different options and we'll go over a scenario where we use each one. But that's pretty much what's going to be going on in every example. Download, embed, load, split. Okay, so now that we got that out the way, let's just run this first workflow. So there's nothing in our Pinecone right now. We're downloading the file right here. As you can see, it's coming back as PDF. We can view it. We can see that it's the correct PDF. We're taking this node, extract from file, to get text out of the PDF. So now we have— this is the actual information we're looking for. And also what comes along with this is information we don't want, like these number things, the info about the document, the version. If you go to JSON, you can see that's what's coming through. And so if you were to just put this in a Pinecone like this, Pinecone node right here is going to be grabbing all of this information. So if we switch to table, we can see that it output 8 items. So it put into Pinecone 8 vectors. And here's the content of each vector. We don't really want this stuff. All we want is this vector with this information, because at this point, if we were to, you know, query this database using RAG, it would be also having to search through this random content that we don't need. So we just want this one, and so we'll talk about how to get just that one. But anyways, the data loader here, it is grabbing JSON, so it's grabbing everything in this JSON that is being fed into it. And then finally, we're just splitting it recursively. So there were 8 different runs here, as you can see, and it was splitting based on what made sense, which that's sort of what the recursive one does. So we've got that out the way, and that was just doing all content. We haven't put any metadata in. And now we'll move on to the next one. Although real quick, let me just go into Pinecone and we'll see what it looks like. So we'll just give our index a refresh. We should see those 8, um, Vectors being put in here now. There we go. So we got our 8 records coming through. And as you can see, the actual text is what is actually the content of the vector. So if we find the right one, um, it was the 7th one, was the 8th one. Okay. So this is the actual information that we really would want to be searching through, um, if we were doing RAG. So that's not how we want to do it. Let me clear out this index and then we'll look at the next example. Everyone just wanted to mention that the workflow that you see in this video with all these examples will be available for download in my FreeSchool community. Link for that will be down in the description. You go to YouTube resources, you would click on the file right here, download that, and then within n8n, you can go up to the top right and just import it from file straight into your n8n instance. And then if you guys are looking to go a little bit further, check out my paid community. I'm doing a deep dive on vector databases. As you can see, obviously Pinecone is kind of the first one up, and then we'll look at Supabase, Qdrant, pgvector. Vector databases are so, so important when you're building out AI automations and AI agents. So it's one of the things that I'm trying to become an expert in. So we've also got weekly We've got a great community. We're doing monthly competitions, stuff like that. And you definitely want to get in here because the price is going to increase at 75 members. So let's get back to the video. Okay, Pinecone is clear. Let's run this example now. As you can see, we're not extracting text and we're loading this as binary. So this comes back once again, it's binary, it's a PDF, it's being loaded into Pinecone. And if you notice, there's only one item coming back. And because we split this into binary, it's only grabbing the actual content we want. As you can see, it took away all the spaces. So this is the first paragraph of the document. It took away all the spaces, but it only gave us one vector. And we did recursive once again, and we made the embedding the same thing. All we changed here is that we loaded binary straight in rather than extracting text. And then we made our loader binary. So go into Pinecone, give this a refresh real quick, and we will see that we just got one back. And then within this one, we are seeing the actual text. But we also, if you notice, got more metadata like PDF information, the title, the pages, the version. So we still are getting that back, but it's not making 8 different vectors and it's not going to slow down our search. So let me get rid of this vector now. We will test out one more example over here, which is the wrong embedding. And if you guys, you know, if you want to take a guess at what's going to happen, nothing's going to happen. It's, you know, it's trying to load it or it loaded it. It's trying to embed it now and it's just not going to be able to embed it because We put this one in as 3 large, I believe. Okay, it's not showing, but I put this one in as 3 large. There we go. And obviously our Pinecone vector store is 3 small. So what's happening is our Pinecone index is looking at dimensions of likely 1536, which is just the numerical representations of the information that we're putting in. And embedding 3 large is probably going 2036 or something bigger than 1536 or more. And so it's trying to fill out the information from this binary, and it's trying to turn that into numerical representation so that later it can be able to go pull things that are relevant or similar. And because the dimensions aren't matching up, it's just trying to embed it and it's never going to be able to. So as you can see, this has just been spinning. I'm probably just going to stop this because I don't want to sit here and wait forever for it to inevitably error. But that's why it's important to make sure that you know your embedding of your vector index and that you're always aligning it. So we're going to stop this one. Okay, now we're moving on to an example where we are getting, putting all the content in. So similar to this first one where all the content was going through after we extracted it from text. Now we're doing that, but we're also loading in metadata. And so metadata later, you can use that to, you know, filter across different vectors to be able to grab them, to reference them later. So in here, when we load it, you can see we added an option, which is the metadata. We added a field document ID, and then for the actual value, we just extracted the document ID. We referenced the original node where we're pulling the file, and that's how we got the document ID. So once again, this one made 8 vectors, as you can see, but at the end of each vector, we got our metadata right here, which is document ID, and then we have that information there. So we'll hop into Pinecone, give that a refresh as well, and we will see that within each of our 8 vectors, and I think, let's see, where's the one with the actual text that we like? It's always different. But anyways, if you see here, now we have the metadata document ID with our document ID as well as the actual text coming back. So let me just delete all this information and then we'll run another example. Okay, next example here. Let's just hook this one up. What we're doing now is we're only getting the text we want. So similar to the one with the binary, but now we're extracting it from JSON. So this one is still going to be loading it as JSON. We're still doing our metadata, which is just the ID. And obviously this is just an example. You can add more metadata. It'll be probably more advanced than this one. But anyways, this is the vector. It just made one and it got us our information we want. We'll refresh Pinecone. And if you're sitting in Pinecone when you run it, it'll refresh automatically. But if you weren't open on the tab, then you're going to have to refresh it. So maybe I'll show that in a future one. But anyways, now we just got our one vector. We just got the text we want back and we have our document ID right there as well in the metadata. So that is good to go. Let's move on to a next example, which is token splitting. So in this one, as you can see, the previous ones we were using recursive character text splitting, which basically is going to— what's the word I'm looking for? It's going to intelligently break up the information. So in this case, the vector wasn't that long, so it was able to just keep it all as one. But let's say you have a huge document, right? It's going to split things up in a way that makes sense so that you can, you know, keep the context of your sentences, keep the context of your documents, your policies, whatever it is. That's what the recursive one is, and that's when it's just pretty easy to set up. So that's the one I use a lot. But there are cases where you need to split up things differently. So in this case, we're using a token splitter. We made the chunk size just as 5, so each kind of word or punctuation is going to be considered a token. This run is going to be splitting things up in token chunks of 5, and we'll just see what it looks like. So I'm going to run this, and then I'm just going to go into Pinecone. So we will probably be able to just see this come through live once it has, you know, split everything up. Otherwise, we might just have to refresh it again. Okay, so I think obviously that just popped up, so Something is happening. Um, query use 6. Let's see what happened in here. So we got 10 items back. Okay, I'm just going to refresh it. So anyways, 10 items back, we can see that we got all 10. And I'm just going to go into NN to actually look at this stuff because it just looks a little better in here. So we got these 10 vectors, right? So document ID colon 123. So that was 1, 2, 3, 4, 5— or sorry, 1, 2, 3, 4, 5. Anyways, it's taking 5 chunks because I think it's also considering these numbers as individual chunks. But as you can see, each one has 5. And if you were to overlap, it would just take like, you know, if it was overlap of 1 right here, we would see this is the, and then the next vector would start with the. And then if this one ended with document, the next one would start with document. So it's going to overlap. Again, we'll show an example of that. But that's how the token splitting works. And yeah, if you have a specific use case where you need to split up that stuff, then you would do token splitting. So let me just get rid of these vectors and then we'll go into the next example. Okay, this time we're doing the character text splitter. So the final option, recursive text splitting, and then, or sorry, token splitting, and then you've got character text splitting. So similar, all of these ones, you'll set up a chunk size and chunk overlap. So that's going to remain consistent. But in this case, what we're doing here is we're adding a separator, if that's something you want to do. So we'll give this a run. And obviously within this example data that's coming back, we have Okay, that was a horrible way to look at it. We'll just go in here. So we have sentences, and each sentence obviously is going to end in a period. So if you wanted to make sure that you weren't losing context of data, you had policies that are going to be all separated by pipes or colons or slashes, then you could split by that character. In this case, we're just using a period. On the left, you can see we've got a period after document, we've got a period after topic. So what it's going to do is it's not going to split it every single one. It's going to just make sure when it hits that chunk size of 50 that if it was to push that boundary, it's going to stop at a period. So all of this makes sense. They're all chunking in the right area because we don't want to lose context. But it made 3 vectors, as you can see, and they're all complete sentences. It's not just going to run off in the middle of a sentence. It's not going to cut off in the middle of anything. And, you know, similar to the idea of recursively text splitting, where you're getting this information back in the way that you know, it makes sense. That's pretty much going on here. And as you can see, we're still doing our metadata here, which is just document ID. So we've got these 3 vectors that came back correctly because we split them with a period. So that's how that one works. Let's move on down. Now we're doing a chunk overlap. So in this case, with a chunk overlap, we are using our character text splitting and we're doing a chunk size of 60. But we're saying here that we want every chunk to overlap by 20 characters or tokens. So we'll give this one a run. It's going to be doing the same thing, grabbing that content, putting it in. We actually got 6 vectors this time. And if we look at them, we can see what came through. So this vector, this is the first paragraph of the document. It talks about, and then the vector 2 starts with, it talks about, and then it ends with the second paragraph. Contains information it, and then this one goes information it. And so you can see it kind of iterates over. It doesn't actually do it for every single one. It doesn't overlap every single one. But I mean, honestly, off the top of my head, I can't think of a use case where you'd want that. Maybe just for extra insurance that no information gets lost if you had a huge chunk size. And also that would probably be, you know, more expensive the more vectors and the more chunking you're doing. But it would be just another extra level of insurance that you're not losing any words, you're not losing context of certain sentences. And so here are those 6 vectors in Pinecone. And now one thing I wanted to really briefly talk about was, is the concept of like updating this kind of stuff. If you saw my video about this, like RAG Done Right, or, you know, RAG Got Even Better, and you're able to— in Supabase is what we were using there— you could go in and delete a document and then you could re-upload it basically. So if you change information, it would go find that metadata ID that you created for that, you know, document, it would delete it and then it would refresh in a new one. And so Pinecone's a little different here. As you can see, we have our metadata, right? But then we actually have IDs for each vector. So let's say we grab this first ID right here, which starts in 6A9. And then we were to come into, let me grab this trigger. We were to come into right here where we're using a different file. So this is just an updated file. As you can see, this is probably not the best way for me to show you guys, but in here we are just basically changing things. So the document ID, we added a 6 at the end. I put updated in all caps in front of different paragraphs just to see how this is working. And so typically you would expect that you could filter based on metadata in order to like change the vectors. But this one has, in Pinecone, you do it by ID. So if I paste in that ID that we were just looking at, It's a little different because we had one document, right? And that one document, we put that into Pinecone and it made 6 vectors. So when we run into the issue of we want to update that document, we would have to grab 6 different IDs dynamically. We would have to figure out how to do that. So it's a little bit different than Supabase. And, you know, another reason that I'm making that sort of more of a deep dive course in my paid community, if you want to check that out, link in the description, we're really going to dive into all aspects of vector databases, Pinecone, Supabase, Qdrant. Pg_vector, all that kind of stuff is the goal, you know, over the next few, few weeks and months here. But if we test that out, I was, you know, having some issues with this. So, okay, so we tested out, we got one item and we got the vector with the correct information that we wanted. But you can see like we, we just embedded it and we didn't actually have to attach like a splitter. So the, what's it called, The ID that we were looking at was 6A9. And if we come back into Pinecone and refresh this, we just need to look for 6A9, which is now at the bottom. And as you can see, the text we got back was just document ID 1234. So still looking into how to do this kind of stuff specifically for Pinecone for the time being. So I'm coming back into this original run where we, this is the, you know, part of the one that we did for vector that we just updated. So we'll run this one again, we'll see what comes through. And this put out 6 items once again. Not really a shocker because that's what the first time, but we will run this and we can check on our information back here. So here's 6A9, still didn't change. Okay, so now when we refresh Pinecone, we have 10 items coming back. Now if we were to look for 6A9. It shouldn't have— oh, I just, I must have passed it. 6A9. Okay, so I actually don't see it anywhere, and maybe that's where it comes into play that it updated. But, you know, if we look around, you can see that it was splitting up, it was doing the, um, the overlap. And so nowhere to be found is the updated in all caps that we want. It actually like went back and redid some of the other stuff. So trying to figure out that information. Um, so yeah, I mean, if any of you guys have, have played around with that and understand it, um, specifically with updating stuff like that in n8n, maybe it's because— oh, interesting, so it created a different namespace because, um, as you can see, you know, typically in here you, you set up a namespace, but there's no other options in here. So still trying to figure all that stuff out, but Like I said, if you guys have any insight, you know, shoot me a message. I'd love to chat about it because, you know, I think Pinecone's a really, really great, great tool for like really, really large-scale vector databasing and, you know, the way you can split it up into different, different namespaces, stuff like that. So I'm definitely trying to figure it out. But anyways, that's all I got for this one, guys. I hope that this was, you know, insightful. These are definitely concepts that are important to understand as far as just the way you're putting stuff in how you can use the metadata to later, you know, look through more information. When you add a Pinecone node right here and you wanted to go to like get ranked documents, you can filter here based on metadata. So then in this place we could put like, you know, document ID equals 12345, and then when you're prompting it for what you're looking for, it's going to make sure it's only pulling stuff back from, you know, that specific metadata. So it's obviously important and something that's important to know because I realized after I was looking into this stuff that some of my first couple videos I probably was putting in way too much information and it was just kind of a waste and probably was slowing down the agent. So that's what I've got for today. I appreciate you guys making it to the end and I will see you in the next one. Thanks guys.