Use Parallelization to Make n8n Workflows Faster & Scalable
Dr. Violet Howard shows how to run multiple items at once in n8n workflows using parallelization. A workflow researching four tools dropped from 51 seconds to 22 seconds by converting logic to a sub-workflow and disabling wait for completion. She covers sub-workflows, rate limits, and error handling strategies.
Chapters
Today I'm gonna be talking about an automation technique that I hardly see anyone talking about on YouTube, and I've actually never covered it on this channel either. It's a technique called parallelization, and it lets you run items at the same time so that your workflow finishes a lot quicker and it's a lot more effective. Understanding parallelization and sub-workflows is really going to level up your automation game. So in this video, we're gonna be talking about what that is. We're gonna see some examples of how you can actually set that up. We'll talk about the drawbacks and considerations, and we're also gonna talk about why and when you would want to set up a sub-workflow, because parallelization is only one of the many reasons why you may want to cut down a big workflow into some sub-workflows. So I don't want to waste any time. I'm just going to run a quick visual demo so that you guys can see how slow this is when you run something sequentially rather than in parallel. So right now we're doing Perplexity research on 4 different tools that are stored in a Google Sheet. That happened pretty quick because Perplexity's API can actually do that in parallel. But now this AI agent is going to process each tool and each research one at a time. So right now it's working on the first tool. As you can see, it just finished up the first one and now it's going onto the second one. And we can tell because if I move this stuff down, you can see it's only done one item. Now it's gonna do the second one. Now it's gonna do the third one and so on. But if we would've had this agent running in parallel, this would've finished up like 20 seconds ago already. So there we go. It finally just finished up and that was only with 4 items. Imagine if you had 10 or 20. So we're gonna dive into this actual flow right here and I'm going to turn this into a parallel running workflow. But before we do that, let's real quick hop into an ExcalDraw so I can explain what parallelization really is. So what parallelization is, besides a mouthful, is that you're running multiple tasks at the same time instead of one after another. And this really doesn't natively happen within NN. We get pretty lucky here because Perplexity's API takes all 4 items and it basically on their server does research about all 4 at the same time. So that's great. But not every API does that. We basically just found out that Perplexity does. But as you can see, when we're calling the chat model, it processed these one at a time and it was very slow. And just in case it hasn't quite stuck yet, pretend that we want to cook these 3 turkeys and they each will take 30 minutes to cook, but we only have 1 oven to use. So without parallelization, we'd cook the first one and then we'd cook the second one and then we'd cook the third one and it would take a total of 90 minutes. But if we're able to use parallelization, which means we would have 3 ovens that can all run at the same time, the whole cook time of these 3 turkeys would only be 30 minutes, and it's just gonna be a lot more efficient. So hopefully that demo shows you guys it's basically a no-brainer in some cases to use parallelization. So let's hop back into NN, and I'm just gonna show you guys how easy it is to get this set up. So in a recent update of NN, they added this cool feature where you can basically select nodes, right-click, and then say, I wanna convert these 7 nodes to a sub-workflow. So if you don't see this option, just try to update your instance of NN, and then you should see it. and what happens is when I click on this button, it's basically gonna tell me to name my workflow. I'm just gonna keep it as my sub-workflow for now. And when I hit confirm, it turns all of that into a sub-workflow and it's already linked up right here and being called by this guy. And so if I click into this, you can see that it's passing in the name of the tool, which is pretty cool. It's already configured. And I can click on this button right here, which will basically take us to that sub-execution. And we should see that everything is still set up and should run just fine for us. But we're not done yet, because all this is doing so far is passing all 4 of these tools, which would be Fireflies, Excalidraw, Loom, and Napkin, into the sub-workflow. And then the sub-workflow over here would capture all 4 and do the exact same thing that we just saw. So what we did is we turned this into a sub-workflow, which is great, but we don't yet have parallelization, because it would still run all 4 through here, and it would go 1, 2, 3, 4. So to make this actually run in parallel, we want all 4 items to run as 4 individual executions and we want them all to run at the same time. So what I have to do is click into the node, and the first thing I'm gonna do is rather than run once with all items, I'm gonna choose run once for each item, which means it's going to call the sub-workflow individually for each item. And then once we've changed that, we have to do one tricky thing where we add an option and we want to turn off wait for sub-workflow completion. If we left this checked on, it would basically send the first one and then wait for that one to finish, and then send the second one and wait for that one to finish. So that would be the same as basically looping through our list of items, and that would be even slower than the way we're currently doing it with sending all 4 items over there at once. So now that we have this configured properly, I'm gonna go ahead and run this one once again. We're gonna go into the sub-workflow and click on executions, and you're gonna see that all 4 items are running at the exact same time, and they're all gonna finish up a lot quicker than the way we previously did it. So we'll go ahead and execute the workflow. It's gonna grab our tools, it's gonna pass them into the sub-workflow right there. I'll click into the sub-workflow and click on executions. And you can see right here, we now have all 4 of them running completely in parallel, and they're all gonna finish up a lot quicker than if the AI agent was doing an analysis on each of them individually. So you guys can see that all 4 of these finished up and it took a total of 22 seconds. We can go into our Google Sheet and see that we now have our features and our analysis on each of these 4 tools. And just to show you guys that it really is quicker, if I go back into the main workflow and we click on executions, and we click onto one of the ones that ran when it was running all 4 at the same time, which I believe was this one, this took 51 seconds. So we cut the time of processing in half, even more than half. And I know it doesn't make a huge difference when we're only running 4 items, but think about if you started to run 10, 20, 30. You're going to save so much time in that automation. And just remember, in the node where you're calling that sub-workflow, in order for things to run in parallel, you have to have it run once for each item and you have to turn off wait for sub-workflow completion. Okay, so it was that easy to see how we can set up parallel runs. Now let's quickly talk about drawbacks and considerations of using parallelization. So initially these 4 came to mind, which were resource limits. So for example, in our flow where we're calling Perplexity to do some research, we may have a limit where you can't do more than 20 per minute or something like that. And if you do run into an issue like that, then you do have to consider looping individually through each item or sending over batches of 20 or something like that. You can't process 100 items all in parallel. Another thing to think about would be error handling. So if you're running 50 items in parallel and one of them errors, how do you handle just that one that errored? And what do you wanna do with the rest of the ones? Or if they all errored at the same time, how do you basically queue them back up or process ones that errored? You've also got workflow dependencies. So in this example, it was completely fine, but sometimes if you have a pretty sequential flow where you can't do step C until step B is done, then you do want to have the logic actually make sense so that if there are dependencies, you're still good. And then finally, a little bit similar to number 2 is troubleshooting and debugging, just because if you have 50 items running as 50 different executions, you will have to sort of be able to find the right executions when you need to make changes. And right now I'm just trying to throw you guys some high-level points. If you want to see this document that I compiled that has way more detail about sub-workflows and parallelization and some stuff like this, then definitely check out my free community. I will link that in there. In the post associated with this video. In the free community, you can also get this template if you just want to play around with it a bit. And all you have to do is come to here, search for YouTube resources, or search for the title of the video up here in the search bar. And like, let's say this was the video that you're looking at right now, there will be the PDF of this document that I'm talking about, and then also a JSON file for the workflow. So the link for the free community is down in the description. Okay, so that's drawbacks and considerations of parallelization. And so clearly you don't always want to go straight for parallelization, but it's really important to understand still to use subflows because subworkflows are really valuable. So here are 4 reasons of why and when to break them up into subworkflows. The first one is repeated logic. One cool thing about N8N is you can build these reusable components. And a reusable component, sort of like this one, where we're going to pass in something to be researched and then an agent to analyze it. This subflow can be linked up to 100 workflows. It doesn't matter. It can be linked up to as many as we want. and so if we know we're creating something that has repeated logic or could be reused, if we package it up as a sub-workflow, we can then call it from all these different places. And if we ever need to go in there and change a model or change the HTTP request or change the prompting, we can just do it in one spot rather than having to go change it in 100 different spots. If we link that up to 100 different workflows, we also have workflow complexity. So if your workflow is getting really large and you just wanna clean it up sort of based on different modules, it's a really good idea to just split it into a sub-workflow. Especially with how easy it is now to just highlight something and convert it to a subflow. Especially if it's not just you working on a workflow, because then you have this whole workflow hygiene thing and everything can be labeled and it will make a lot more sense to someone else who needs to come in there and debug or improve it, whatever it is. Obviously we have parallel processing, which is what we just talked about, and then we have error handling and recovery. So maybe you can have things always go into a subworkflow if there was an error, and then that can just basically handle all the notifications and whatever else you need to do. but it's also way easier to tell which sections of your workflow are erroring if you have different components split out into different flows. And then you can just basically understand, okay, you know, this research section of my workflow is failing a lot more than the data manipulation section. And now you know a little bit quicker what you need to work on and what you need to refine. It just helps keep everything split up and separated and really clean and organized. But once again, you can dive a little bit deeper into breaking up workflows in this document that I have prepared for you guys. And just to show you guys one more quick example of how this node works, let's say we have this flow, which kind of looked like this, and we decide, okay, this section right here is gonna be a really good reusable component. So we're gonna turn that into a subflow, which we did right here. But let's say we're hitting rate limits on our Google Sheet node or our Perplexity node. So we don't want to process them parallelly, but we still want to keep that subworkflow to clean it up. All we'd have to do is we could come into our main workflow, we could click into here, and then we could just turn this on, which says wait for subworkflow completion. and this basically does the same thing as if we added a loop over here and we were just processing all 4 of these items one at a time. So just to show you guys that, I'm gonna go in here and I'm going to clear out the data. I'm going to set all of these statuses to ready so that they'll actually be pulled into the automation, and then we'll come back in here. I'll hit save, and then I'm going to execute this workflow. It's going to send all 4 over here, but you're gonna see it's gonna process these one at a time. So if I go into the actual subflow and I click into executions, You can see what we have is just one of them running right now. That's it. Once this one finishes up, it will then trigger off and fire the second one. So hopefully you guys can see how this is basically turning these into batches of one and just sending them off one at a time. So you can see the first one just finished up and now it's processing the second one. So this is going to take about 20 seconds for each one. So it's going to take a minute, which is a lot longer of an execution than having them all finish up in 20 seconds at the same time. but this is a way that we can protect ourselves from hitting certain rate limits. So if you guys found this video interesting and you wanna dive deeper into different automation techniques that are a little bit more advanced, then definitely check out my paid community. The link for that is down in the description. We've got a great community of members who are building with AnyDesk every single day, always sharing their resources and sharing the challenges that they're running into. We've got two full courses in here. One's called 8 in Zero, which is perfect for beginners. It's like the foundations of AI automation. And then you move from that into 10 Hours to 10 Seconds, where you learn how to identify, design, and build time-saving automations. So if any of this intrigues you and you want to be surrounded by over 2,000 like-minded individuals, then definitely check out the community. And I was able to do that entire school plug while this was still running sequentially, which just shows how much quicker it is to do parallel. Anyways, that's gonna be it for this one, guys. If you appreciated the video or learned something new, please give it a like. It definitely helps me out a ton. And as always, I appreciate you guys making it to the end of the video. I'll see you on the next one. Thanks, guys.
