Beginner's Guide to Workflow Evaluation in n8n (Stop Guessing!)

Today we're gonna talk about something pretty boring but super, super important, and that is workflow evaluation. So I'm gonna start off by talking about what it is and how it works. Then we're gonna get into N8N and I'm gonna show you guys me actually running some evaluations. So let's get started. All right, like I said, today we're talking about AI workflow evaluation and specifically within N8N. I thought it would first make sense to talk about what is evaluation. In my mind, the simplest way to put it is validating your hypothesis with objective proof. Think back to when you were building an AI agent or workflow and you weren't happy with the results. What you do is you think about, okay, if I do X, it's going to make the output better. And so that's your hypothesis. You go ahead and you make that change, you run it again, and you basically subjectively judge, is it better or worse? But by running evaluations, we can actually see data and we can see proof if your change validates your hypothesis or not. So now that we all are on the same page about what evaluation means, let's start to look at how this would actually work in NN. So for this example, what we're looking at is a tagging agent. It will receive an email, it will look at the subject and the body, and it will use its AI brain to basically tag the category and priority of an email and then do whatever it has to do later. And so what we're looking at here is a test dataset where we say, okay, here are 6 really good examples of an email that came in and where they should be categorized and marked as far as priority goes. So what we would do in our evaluation is we would feed through these 6 inputs and we would basically validate, are they answering the expected category and expected priority correctly? And when we run these through an evaluation, we can actually see each of those 6 test cases, and we can look at how many tokens were consumed, how long it took in, you know, time, and then we can see if the priority or category was right or wrong. And then because we're able to identify patterns like, you know, maybe when it's shipping inquiry, it's actually being labeled as support or something like that, we can go ahead and use that data of what is actually happening to go ahead and fix the prompt. And then we fix the prompt, run it through again, and we see if we get a higher accuracy score. So in a couple minutes, we're gonna hop into NN and actually look at this exact example. But real quick, I wanted to get some of the more boring stuff outta the way real quick, which is just kind of laying a good foundation about the mindset of AI workflow evaluation. So evaluation has been around for a long time, but why is AI workflow evaluation fundamentally different than code evaluation or other projects? Because AI LLMs are kind of known as a black box because we don't know exactly what is going on and how it gets to its end result. Meaning if you put in 100 of the exact same inputs, you'd likely get 100 different variations of outputs. Whereas in a traditional workflow with no AI or a code script, you could put in 100 of the same inputs and you should get 100 of the same outputs. We also have the probabilistic nature. Output is shaped by probability, temperature, and context, and there's so many different levers that we can pull to train the way the AI actually behaves. We also have evolving models as new ones come out. If they're cheaper and faster and people say they're the best, you're gonna switch it out. But does that new model actually increase the performance of your workflow or not? And there's other things that we want to optimize for beyond just accuracy and consistency. We also have stuff like bias and cost and time. So because there's so much to take a look at, there's a lot of noise. The key metrics to really be thinking about when you are running evaluations and you want to track would be performance, reliability, efficiency, and quality. And by the way, I'll have these slides in my free school community. The link for that will be down in the description. You can download that, and you can also get the two workflows we're going to look at. So you can get all the resources shown in this video in the free school, just in case you thought I was going a little too fast. And now moving on to something that is probably the most important part of evaluations— well, second most. The most important thing we'll talk about at the end. But the second most important thing is being able to isolate variables to actually understand what's going on. I don't know how long you guys have been watching my channel, but something I've talked about before is reactively prompting your AI agents. Meaning if you aren't happy with the output of your agent, you want to change one thing in the prompt at a time rather than if you get in there and you add like two paragraphs and all of a sudden your agent's performing horribly, you don't know where in the two paragraphs you broke the agent. Or on the other side of things, if it makes it better, you don't know what specifically made it better. And this is honestly even more important when you're doing evaluations because let's say you're getting, you know, 70% accuracy and then you change the prompt and the tools and the chat model and all of a sudden you're getting 85% accuracy. You don't know which one of those had the, the most effect on the actual increase in performance. So what you need to do is treat it as a scientific experiment to validate your hypothesis. Change one thing at a time, whatever that is, keep all of the test conditions consistent So keep all of the variables not variable, keep them consistent, and only change one thing at a time. And document what changed, why you changed it, and what the actual result was. This allows you to actually test different things and optimize for the best model possible. Common variables that you should change and test different things would be the prompt wording, would be the actual LLM itself, and then within the LLM, you have different things like temperature, top-p, all this kind of stuff, the workflow design, and data preprocessing methods. So in other words, kind of the context that is being fed to the agent or to the AI before it does its action. And now this is the most important part of an evaluation is the dataset. The purpose of your evaluation dataset is that it has to be the ground truth. This is basically what we're training our AI workflow to perform like. So it has to be good. And what does good data look like? It's accurate, obviously, it's consistent, it's comprehensive and representative of lots of different scenarios that may go through the system. It covers edge cases and it has to be large enough for statistical significance, to have confidence in it. And some examples of where you could get this type of data would be historical high-quality tickets, expert responses, high-performing marketing content. And ultimately, you're probably going to have to work with the subject matter expert of whoever was manually doing the process beforehand or overseeing the manual process. They are the ones that should be able to provide you with all of that good data. So for example, if you're building a system to make LinkedIn posts, you would want to go grab all of the best performing LinkedIn posts and use those as data that you want the AI workflow to consistently output stuff like those posts. And last slide here, then we're hopping into N8N. Why data is the core of success. Well, it's pretty obvious. Good data would be, you know, good out, good evaluation, good AI behavior. Bad data is you're literally just training the LLM to be worse. It would be like every time, you know, a kid cusses, you just start laughing and you give them a cookie and you say, good job. That's just like not the way you should do it. And anyways, when you think about size, ultimately more data the better, unless it's bad data. But if you start small and you're trying to get some early testing, 50 to 100 examples is good. I'm not gonna do 50 examples today. I'm gonna do like 6 or 10. But just, I wanted to throw this out there to show you that you need more than just 10. When you really are starting to think about pushing this into production readiness, you want to have upwards of 250 to 750 examples where you can actually get statistical significance and cover a wide range of scenarios. Now take this with a grain of salt because the amount of examples you need definitely depends on the actual system and the variety of inputs it could receive, but just kind of rough ballpark figures. And then mission-critical, high-risk systems, you want to go above 1,000, get as much data as you can truly. Um, something else to think about is if you were to think about right now, I need to source 1,000 examples. That would kind of suck. But if you're thinking about getting good data and collecting data over months so that when you're ready to test the system, you have it, that's going to obviously make the workload less. But also when you do that, you're collecting data, like I said, over months. So you're not just grabbing, you know, all the data from the past week or something where that wouldn't be as holistic as when the system is actually gonna be in production and facing this type of real stuff. All right, so now we're actually in n8n and we're gonna look at how we do an evaluation flow. So I know this workflow may look intimidating, like right off the bat because there's so many different nodes. These are all of the new evaluation nodes in NNEN, and I'm gonna explain exactly how they work. So if you were to come in here and type evaluation, you can see there's 4 new nodes. We have the trigger, which is over here. We have check if evaluating, we have set metrics, and we have set outputs. So what happens is when we do this trigger, it's gonna pull in our dataset, which in this case, we're gonna use this dataset of 6 examples. With our expected category and our expected priority. So it's gonna pull in those examples. It's going to feed them through the AI model, which is supposed to tag them, you know, the different categories and the priority. It's going to check if we're evaluating or not, because you could still have a typical flow. So if a Gmail came in, it would go down the normal branch and you could set the logic. But if you're evaluating, it'll go up here, set outputs, which just means it's gonna write the AI's answers back into the Google Sheet. And then it will set the category and the priority, which we will be able to check in the evaluations tab. So just to show you guys how that looks, I'm going to do one run through here, and then we're going to actually do the evaluation. So just to pull in the dataset, it's going to pull in the first row, the AI is going to tag it, it's going to go up the evaluation branch, and then it's going to write the outputs back to the Google Sheet. I'm just going to stop this real quick, and then it would have set the category and priority. So if I go into our dataset, you can see that that first run it got the category wrong because it was billing issue rather than billing, and the priority was right. And so obviously this is just a mistake of like wording. And the reason it did that is because this AI agent has no system prompt. So just wanted to show you guys like a very, very rough baseline of how this works. I'm going to clear this out and what we're gonna do is check what chat model we're using. We're using 4.1 mini, and then I'm gonna go to evaluations and just hit run test. And so this is basically going to send those 6 different items in our dataset through this model. It's going to collect an average of how many tokens were used, prompts total, and then also how long it took. And then it's going to give us an average of the priority and category score correctness. So looks like that already finished up. If I clicked into it, you can see it processed all 6 and you can see which ones, you know, were right and which ones weren't. And so let me just go back to the actual execution. You can see that we got 67% of priority correct, but 0% of category correct. We could also look at different figures over here, of course, like our time and how much it costed us. But this is mainly what I'm focused on right now. And so you would see this and you'd go into your sheet and you'd see, okay, all of these categories are wrong. What we could do is work in a quick fix in the prompt because it seems like it's just messing up because it doesn't know the actual words. So what I'm going to do is just copy these words right there. I'm going to go back into our workflow, into the editor, and I'm just going to add a really quick system prompt into the agent where I am just going to say, here are the categories to choose from, and I'm just going to paste those in like that. I think this alone will make it much more accurate. So we're just going to go to evaluations. I'm going to hit run test once again. It's going to pull in that same dataset and it will overwrite whatever already exists in here. And you can see it already got that first category right. You just got the second category right, got the third one right, it got the fourth one right. And I'm assuming it's gonna be 100% accuracy on category, but then we're still messing up on the priority score. So just finished up, we can see the evaluation now. We got 100% correctness in the category, we got 67% correctness in the priority. And now we would just be able to keep making one tweak at a time and and know exactly why what we're doing is working. And so yes, you have this data here, but what you don't see is you don't see what you changed unless you clicked into it and looked on all the executions. So what I would recommend is when you start running these evaluations, you would maybe in your dataset have another tab where you could say run number 1, what did I change? And then what were the results? And then you can just basically keep stacking all those runs and evaluations to see which ones gave you the best scores. So I hope you guys can see how this would be really helpful in creating a workflow and then basically deciding, okay, well, which model do I want to use? You just test like 5 of them, see which ones have the best accuracy and were also the cheapest, and boom. I don't want this video to go too long, so I won't really dive into what's actually going on here. But basically in these metrics, I'm just setting it as what the expected answer is, what the actual answer was, and then that's how it outputs either 1 or 0 if it was correct or incorrect. But what you can see here is that there's lots of other metrics. We just did categorization, which means explicitly, do these two things match? But what we want to do is sometimes we need to use like correctness or similarity or helpfulness, and that's where you actually need to use AI to judge the correctness or similarity. So this next example that we're going to look at, this one actually uses AI to evaluate. So similar type of scenario, we have a, agent that's gonna read an email, it will look up in its vector database FAQs and policies to answer that email, and then it would respond back to them. So what we're gonna do is we're gonna look at this second dataset that has subject and body, it has expected answers, and then we're gonna get the actual answer evaluated on how close it is to these expected answers. And so before I actually kick this off, I wanted to show you guys one thing real quick because what you can do is use the actual Set Metrics node to do AI-based correctness. And it will give you this full prompt about like basically you have your expected answer and you have your actual answer and it will output a score 1 through 5 over how correct it is. But right now, I'm sure by the time everyone's watching this, it'll be fixed. This node is basically broken and it was giving me these errors. Let me just show you guys. It was giving me these errors where it would say the Set Metrics node is not configured properly. Even though I knew it was. So all I did is I took this system prompt and I basically fed it into this agent where I'm giving it now the expected answer, the actual answer, and the exact same system prompt. So it's really the same exact thing going on, and then I'm just passing it into a different metric setter. And I know this all may seem a little confusing. Just watch how this plays out. You can download all of these workflows I'm showing you today and my set example datasets from the FreeSchool community if you want to play with it yourself. That'll probably be a better way to actually understand what each node's doing. But I just wanted to show you guys this little workaround because this node wasn't actually working. And you know how we talked about keeping things consistent? I know some people may say like, hey, but you're using an AI model to evaluate, blah, blah, blah. All you have to make sure is that you keep this AI model the same every single time. That's where you'd run into trouble, is if you were changing like the prompt, but you were also changing the model that you used to evaluate. So as long as you keep this consistent, I'm not going to be concerned at all that we're using AI to evaluate. If you watch my GPT-5 video, you may have seen a preview of me actually doing this. I was testing GPT-5 and GPT-5 Mini and stuff like that. So this first run, we're going to do GPT-5 Mini. I'm just going to go to evaluations, hit run test, and we'll let this finish up real quick. And then I'll show you guys what we're looking at. Okay, so this run just finished up. Looks like we had an accuracy of 3.5 out of 5. We can click into the run and we can see individually each of the runs. Like we can see number 3 was not accurate at all. But the first one was super accurate. And what's nice is we can see like over time if our workflow has gotten better or worse. But keep in mind, this doesn't really show us what was changed in each one. So it's still very important to track on your own in like a Google Sheet or something. But let's say for some reason we were not happy about how long this took. So we have an hypothesis that if I use Google Flash, it will finish a lot quicker. So in order to validate that, I would just go into the editor, I'd come in here, and I would just want to go to some sort of Flash model. So we'll do 2.5 Flash. We'll save that, go back into evaluations, and now we're gonna run another test. So the hypothesis is that Flash is going to be cheaper and hopefully more accurate, but we don't actually know. So now we are going to, oh, and faster. That's the whole reason I said it in the first place. So now I'm gonna let this finish up and we'll compare the execution time and the accuracy. Okay, so that just finished up and I'm actually really impressed. It got 4.3 on the accuracy, so much higher than GPT-4. 5 Mini, and I know for a fact it was cheaper and it was about half the time. So I don't think speed alone in this use case justifies making the switch, but because the accuracy was 0.8 higher, which is pretty significant, then that makes this one a no-brainer. But we were able to objectively see that rather than just having a hunch. Anyways, I hope that this video has been helpful and showed you how you can use this evaluation feature for not just like categorization metrics but also for accuracy metrics and stuff like that. You You'll be able to download this workflow, this workflow, as well as the two different sets of test data right here that we saw in the video. You'll be able to make a copy of all of this stuff and download all of it for free, as well as the actual slides that we were looking at earlier too. All you got to do is go to my FreeSchool community. The link for that will be down in the description. And once you get in here, just go to YouTube resources and you'll be able to find this YouTube video, or you can search for the title of it. It'll be in here, I promise. And then all of the resources will be pretty much in that post associated with this video. And if you wanna dive deeper with N8N, then I definitely recommend to check out my Plus community. The link for that will also be down in the description. We've got a great community of members who are always sharing what they're doing and what they're building with N8N, sharing their challenges, sharing helpful resources. We've also got a classroom section with two full courses right now. One is Agent Zero, the foundations of AI, and then 10 Hours and 10 Seconds where you learn how to identify, design, and build time-saving automations. So I'd love to see you guys in these communities, but that's gonna do it for today. If you enjoyed the video or you learned something new, please give it a like. It definitely helps me out a ton. And as always, I appreciate you guys making it to the end of the video. I'll see you on the next one. Thanks, guys.