The Best RAG System On YouTube (Steal This!)

All right guys, super excited to share this new RAG system with you today. It uses a combination of N8n, Supabase, and Postgres for memory, and it is the best one that I've shown yet on this channel. So it's going to be watching a Google Drive folder, and then when you upload a file, it's automatically going to figure out what type of file it is. The respective logic is going to take place for that type of file. It's going to put it into your vector database with relevant metadata like file ID, the name, when it was created, when it was last modified, stuff like that. Then we have another workflow that's watching those files for updates. If they're updated, it's gonna delete the original record from Supabase. It's going to change the version history from v1 to v2 to v3, and then it's gonna re-upsert them into Supabase with the relevant metadata. If you enjoy this type of content, please consider giving it a like. It really helps me out. But without further ado, let's get into the video. Okay, this is the workflow we're gonna be getting into today. We have 3 main parts. So up here in the yellow, we have the RAG agent. We're not gonna spend too much time here. If you want to learn how to set this up step by step as far as the Postgres and the Supabase, go check out a previous video. I'll link it up top right now, and that one will walk through that kind of stuff. So we've got the agent that we'll be talking to for RAG. Then we have this green section, which is watching for files that are created within the folder. We have a trigger here. It's going to loop through, download those files, figure out what type of file they are. It'll go through respective logic to extract the text, um, you know, optimally for that type of file. and then finally it'll put into Supabase. And then when we are loading the information, we're gonna load up all, all different types of metadata. We'll get into that, um, when we're walking through an example. And then finally we have in the blue the same folder, but we're watching for files updated rather than created. So this is going to, obviously it's gonna grab that file ID so it can delete the original record that we just uploaded in the previous workflow. It's then going to set the version. So in the metadata we'll have version 1, version 2, version 3, and so on and so forth. and then same exact logic of going through, figuring out what type of file it is, putting it into Supabase with our metadata. If you guys wanna download this workflow, go to the FreeSchool community linked in the description, click on YouTube resources, and then go to the post associated with the video. You'll be able to download the workflows right away. Then if you're looking to take your learning a little bit farther, check out the paid community, also linked in the description. We've got a great classroom here with, with deep dive topics. I'm doing one about vector databases right now. This resource library is gonna be continuously updated every week. We've got a really great community in here. All your questions are guaranteed to get answered. We also are doing about 5 live calls per week. Um, so I'd love to see you guys in here. Finally, if you're interested in me building out this type of stuff for your business, or you're interested in AI consulting, then check out my website, link down below, and book in a call. Okay, anyways, back to the video. Real quick, just wanted to give a shout out to Victor from the FreeSchool community. He helped me out with this one, so really appreciate that, Victor. But let's get into this actual build. So we have our agent, we have our Supabase, table, which is just called documents in this case. And real quick, we're just going to chat with this guy, ask, um, what projects do we have in the database? And this is basically going to say we have nothing. So this is kind of just proving that there's nothing in there. We're going to upload a project brief and then we will be able to chat with it. So there's no projects in there. Now let's add something. So, um, looking at this first logic, as you can see, we have two triggers. This one's deactivated because when there's two triggers in the same workflow and you hit test workflow they can get confused. So we're just deactivating that bottom one for now. Um, okay, so I've got Project Brief for Project Wheel. I just got some information in here. We are going to take Project Wheel, drag it into our folder called Projects, and that is the one that is being watched for right here. As you can see, it's called Projects. We'll hit Fetch Test Event to make sure that we're getting the right project back, and we should see the name of it over here, right there, Project Brief. Project Wheel. Okay, so let's hit test workflow and watch this stuff happen. So it's going to download the file right now, and then it's going to figure out, okay, this is a Google Doc. From Google Doc, we turned it into a text file. We did that right here. We extracted the text, put it into Supabase. So let's walk through what just happened. Um, okay, so first thing we're do— we're doing is just setting our file ID and our file type so it's easier to reference for later. So file ID is the actual ID of the Google Doc. So we see this one is— it ends in WOL, and we come into here and we can see that right here. Or actually, you guys can't see it, it's in the URL, but this file ends in WOL. So that's just the file of the Google Doc that we're looking for. And then we're setting the file type, which as you can see was basically a Google Doc. So from there, we're feeding in the file ID right here so that the Google download or Google Drive download node can actually download the correct file. It's going to be dynamic, obviously. And here's one thing that we're doing that's kind of important. So within here, we're setting the options of, you know, configuring this node. And we're basically saying, okay, if anything comes back here, there's a Google Doc, turn it into a text file. If it's a drawing, turn it into a JPEG. Slides, turn it into Microsoft PowerPoint. And then Sheets for CSV. And so now instead of getting back that Google Doc, we're getting a text file. So we can view it right here and it's just the plain text and it's being stored as binary. So from there, we feed it into the switch node, which obviously we have different options. We're basically setting the routing rules. So if the file ID is this, it's going to go that, that way for PDF. And then all these other options, of course. And if you want to take a look for this, I know I'm going to go kind of fast through these examples, but the workflow will be available for download in the FreeSchool community. Link for that will be down in the description. So you'll be able to just grab that, plug and play, and then you will be able to— oh, then you'll be able to just look at what's going on within each node. So anyways, from there it goes down this path because it's a text file. We can see that when we're doing a text file, all we're going to do is extract from text. PDF, extract from PDF. And then Excel, we're doing some aggregation summarizing. And then for Word docs and sort of Windows docs, we are going to convert them. So from there, we are uploading that into our vector database. The only thing really that's fancy going on here is that we are setting the metadata. So as you can see, we have all this kind of stuff coming through. So the first one is the file ID. Of course, this is important so that later we can feed the file ID into Supabase so we can delete the doc and then upload something else in its place. We have the version. Like I said, this later will change. We have the creator, created at, last modified. We have the folder path hardcoded in there, and then file name and file extension. So So that's what's going on within the metadata. So as you can see, this output 2 items. So we should have 2 vectors in our Supabase. If we go and check here, we can see, there we go. They just popped up. So we have project brief, project wheel. We'll click into here and we can see some metadata. So what we're looking at is the creator, obviously the file ID, we see the version and then, oops. And then other stuff that we obviously configured in there, right? That's all that's going on in the metadata. And now what we're going to do is, since we have that information, we are going to first of all chat with our agent to make sure that it's getting it right. So in Project Wheel, let's just ask for some key features of this project. So what are some key features of Project Wheel? While this is loading off, I'll just say we're going to be going even deeper with this sort of RAG system. I'm doing a deep dive right now in my paid community about, about vector databases and RAG. So anyways, here's some features of Project Wheel: data integration, automated syncing, customizable dashboards, and data cleaning. And as you can see, it got that all right. So that's perfect. One more thing I'll show is that we have our chat history. This is the Postgres that we're using. If we go down here to this most recent entry, we have some key features. So this is what the AI responded with. Let me actually just make this a little bit bigger. So here's the key features that AI just responded with, and then here's what we just asked, which was, what are some key features of Project Wheel? So that's sort of how all this memory is being stored in Postgres rather than if you're using a Windows chat buffer memory. All that, all this kind of stuff is going to be stored in NN. So that's kind of how the Postgres and Supabase work in conjunction. Okay, so from there, let's test out this workflow, and let me make this a little bigger so it fully captures everything. Okay, perfect. That was going to bother me if not. Okay, so let's deactivate that node and turn this one on. And now all we have to do is come into Project Wheel. We will change something real quick. So let's just change, um, let's just change the name, keep it real simple. Project, um, Butter. I don't know why butter is the first thing that just came to mind. That's probably not good. Anyways, project butter. Um, we'll go back into the folder and make sure that it gets modified. Okay, so 3:12, um, that means that we just modified it and it has seen this change. So we'll come into the, um, file updated node. We'll hit fetch test event. And, you know, hopefully this one actually registers that that was an updated file. So We'll test this workflow, we'll see everything happen. Um, so right now these record IDs were— okay, so it just got deleted. As you can see, it got deleted. In here we're setting the version, so the previous version on this side, as you can see, was v1, and now coming out is v2. Downloads the file, it's still a Google Doc, does the same thing with text, and then we're uploading the metadata over here, which as you can see right here, the version now is v2. So let's hop into Supabase. Is this going to automatically refresh or do I have to refresh? Okay, I'm gonna refresh. We will see we have only 2, so it didn't duplicate them, it didn't just delete them, it deleted them and then put in the new information, which as you can see right here, Project Butter. Perfect. And then in here we should see v2, so we have the version. Um, also what's cool to see is that we see that it was originally created at this time. I'm having trouble understanding what's going on here. 22:02 and then 22:12. So as you can see, it's changing. It knows that it originally was created at that time and now we've modified it at this time. It's still keeping everything else pretty much consistent, but that's how that works. So, uh, if I was to come into this guy and now we ask about Project Butter, what are some No, I'll just say, what's our budget for Project Butter? And hopefully that works because, um, I changed the name of Project Butter, but also within this project wheel. Okay, so that's what I was worried about. So we only changed it here, but other places it's known as Project Wheel, and up here it's known as Project Wheel. So maybe it wasn't the best example. I probably should have changed some actual information that came through here. But that's not really the point of the video. As you can see, it got updated here. So, um, yeah, it worked as it should. I just didn't really think through the— what I wanted to change because right here it's still known as Project Wheel. So I guess we'll just do one more thing real quick. So previously we're saying that this kickoff meeting is December 1st. Let's change this to December 15th. Okay, so our kickoff meeting is now December 15th. Go into our Drive. Okay, this just got updated. It knows that we modified it. Let's go back into our agent and run this again. So we will see now it will go to version 3. So we'll hit Fetch Test Event. I don't really need to do this, I could have just triggered the workflow. We'll test the workflow. We'll watch it delete. Um, the reason I'm limiting— I'll show you guys in a sec also once we just make sure that this is working. So we should have IDs 56 and 57 in here now. Oh, I guess it already did it too quick for me. Okay, yeah, it did it too quick for me. So v3, as you can see, and now we can come back into our agent and ask about the timeline of Project Wheel. Okay, what is the timeline for Project Wheel? It should understand that and we should be getting back December 15th is the kickoff like we did. So December 15th, December 18th, and January 12th. Okay, so now that we see how that's working and we see that, um, you know, we're getting the version change and the modification date changed, let us look at what's going on. Okay, this is the execution of the v3 project wheel that we just did. So we're downloading the file, we're grabbing the file ID that we set here, of course. So we set the file ID, the file type, and then within Supabase we had to do sort of a little filter to make sure that we're deleting the correct item. So as you can see, the operation is delete. We're deleting a row within our table called documents, and then we're selecting a string. And this is kind of the filter we're doing. So it's metadata, um, and then the dash with two greater than signs kind of just signifying, okay, within the metadata, what are we looking for, which is a file ID. So back in here, this is our metadata, right? And then within the metadata, we have, we have right here file ID, and then we have the file ID. So we're just defining file ID equals like dot, and then within two asterisks, we're doing the actual file ID, of course. And so then once we do that, what we get back is two items because there were two vectors, right? So in this case, it was only two vectors. If it was a longer document and maybe depends on the way we were splitting it and, you know, chunking it, obviously, but it's likely going to be more than just one vector that's associated with a single file. So in that case, it would return like maybe 15 items. And if you did that, then it's going to send like 15 items through the rest of this logic. So that's why we just limited it to 1. So no matter what, it's going to grab the first one, which always is going to have the previous version, the previous file ID, all that kind of stuff. So we're fine with that. From there, we want to set the version. And so my first thought was, okay, let's do a code node just so every time it's going to iterate through, it's going to increase the version by 1. And when I tried that, as you can see on this left-hand side, version is a string. So you can tell obviously right here there's an A, so it means it's a string, which is just a word. But if you look up here, like lines is 40 and 1, and there's a little hashtag meaning that this is a number. So just a different variable type, variable type. So when I tried to do it with a code node, it wasn't updating the version accurately because You can't do math on a word, right? So I realized, okay, well, maybe I could try to, in the code node, convert this to a number variable type, and then it could do it. But I was like, okay, well, we might as well just grab a 4L mini, or you could probably even use like even, even a cheaper model. 4L mini is pretty cheap. But then I just basically said your goal is to take the incoming version number and add 1 to it because it's going to be able to understand what's coming through. So the example, if you get v1, you're going to get v2. If you get v4, you're going to give out v4 or v5. And I said always output it in a field called version, just so when we're referencing it later for the metadata, it's consistent. Um, and then I just gave it the incoming version number down here. So I just dragged that in there and that's all I did within this node. So once we have the version, um, and we still have the original file ID, right? So we're getting this kind of stuff, but then we can reference the file ID from back here. Sorry, it keeps jumping around when we're in the execute or yeah, execution. Then we just do the same exact thing of downloading the file. We're going through the route, of course, and then we're inserting it into Supabase. And then within this Supabase, um, or sorry, not Supabase, within the document default, within the data default loader, um, we're setting the metadata. And everything's the same. The only thing we changed, obviously, is that the, the version we wanted to reference, um, whatever we got from that OpenAI node, that's called setVersion. We're just grabbing the message content called.version. And so then here's what's getting put into our, um, Supabase. As you can see, the version has been updated to v3, and then that's basically it. Okay, sweet. So now real quick, let's just deactivate this one and turn this one back on, and we'll just show like a PDF running through. So I'm going to go to our Google Doc, which was Project Wheel. I'm gonna download this guy as a PDF, and then we're just going to drag this into our projects. So we got that in here. We are now going to just test the workflow, and we should see that it's being downloaded. Obviously it's a PDF, and as you can see now, it went to the router of a PDF. We extracted the text from this binary, and we got the same exact information back. And now that should also be in Supabase. As you can see, it just popped through. And this is Project Butter, and this was a version 1 because it was a new thing that we created, right? So as you can see, it's also the file extension PDF, whereas the first time we did Project Butter, it was an application Google Doc. So yeah, awesome. That's how that works. I didn't want this one to go too long. I kind of just wanted to give over— give a brief overview of what's going on here. Now expanding off of this, right? So like I said, in my paid community, I'll probably be building off of this and talking about making things a little more production ready. And one thing that I'm starting to play around with is how would you— let's say we came in here and we deleted this, right? We deleted this file, we no longer want this project. Nothing's going to happen. It's going to stay within our Supabase because we haven't set up any logic to handle deleting stuff. And right now it's no big deal, right? Because we know these two were the PDFs from this path and we can just delete them. But when you get to the point where you have multiple documents and a document has 10 or 15 vectors rather than just 2, you can't be doing that. So, um, basically the logic that we're going to be going through is we're going to be running one, one flow that's going to grab all of the file IDs, right? It's going to grab all the file IDs that are in your Supabase, and then we're going to have another one that's grabbing all of the file IDs that are within this drive. And then you're basically going to merge those together and check which ones don't have matches. And then any of them that don't have— that can't join, we will feed into a delete Supabase node and we'll feed those file IDs into there. And this would be some sort of scheduled trigger. It could run once a day, it could run every few hours just to check. It's basically just comparing, right? Um, and yeah, so that would be a really cool way to pretty much really make this thing automated. But for now, that's all I've got. Um, like I said, if you guys appreciated this video, please give it a like. It really helps me out. The link to download it will be in the FreeSchool community down below. And, um, that's gonna be it for today. So really appreciate you guys making it to the end of this video. Um, I'll see you guys in the next one. Thanks.