CrewAI Flows Made Super Easy

All right, so this video is gonna be real quick, but the gist of this video is to show you how to use CrewAI Flows. If you've been building AI agents using CrewAI, you might have heard something called CrewAI Flows, but maybe you decided not to explore it because it felt a bit complicated, a bit advanced, right? So this video is to help you understand it. Essentially, that's what it is. So I'm going to break it down really easy with a quick analogy, and then beyond that, I'm just going to show you a quick notebook that you can literally just Copy, and I'm going to share the link in the description. Run it with me, add your API keys and just run it to help you understand exactly what's happening. What if, what's the flow and why do you need to use it essentially? So let's go ahead and get started. So real quick here, we have a notebook. It's CrewAI Flows 101 notebook. And I'm going to walk you through what all these steps mean, what exactly we're building here, but to just kind of get you started off, we're going to be building a dynamic content creation flow essentially. So let's say you have a piece of, I don't know, documentation, a bit of research paper or something of the sort, and you want to convert that to 3 different content types, or you just want to research something specific and create content out of that. So let's assume the content that you want to create, one of them is a newsletter, another one is a blog or a LinkedIn post, for example. So how do you then decide, you know, how do I route to which content depending on the day? Maybe today I just want to create a LinkedIn post about this specific research paper that I found. Well, typically what would you usually do is probably build a crew specifically for running the blog generator, one for newsletter, one for LinkedIn post generator, right? So that concept is still there, But let's say today I just want to, you know, I want a LinkedIn post. I don't have to go combing through all my agents in my directory or somewhere to figure out, okay, here it is. This is the one I need to run and then run that individually. What if you just had one system where you just go to that one agent that you have, you give it a link to that paper that you want to create a LinkedIn post about, for example, and tell it, you know, LinkedIn post, here's the link, go create a LinkedIn post about this paper. And it just goes back on the background and figures out Well, this is the research link. He wants a blog. He wants a LinkedIn post. It routes that task essentially to the LinkedIn post crew specifically that generates a LinkedIn post and that handles that. If it's a newsletter, it routes it to the newsletter. If it's the blog post, it routes to the blog post crew. I hope you get the context here. But I think to help you even understand a bit more, I like to think of CrewAI flows in the sense of a company. Usually a company has The CEO, but then there's departments within the company. You have engineering, you have marketing, you have sales, you have all the other departments within a company. Assume those individual departments are crews. And in crew, a crew is made up of a couple of agents that have specific tasks and specific tools and a goal to accomplish, right? They, they do a specific job. So same thing with a company. If you go to something like engineering, you have different people in engineering, right? You have software engineer, you have a data scientist, you have a data analyst, you have a PM. So many people in there. Each of them has different roles, have different roles, have different objectives that they need to do, the specific tasks. And they also have specific tools that they need to use to get their job done. So for somebody like a database manager, they need to have access to the databases for them to be able to do their job, right? They need to have access to Wi-Fi. They need to have access to a laptop. Those are the tools that they use to accomplish their job. Same thing with a crew. A crew has an agent, which you can think of it as the body, the person actually. And then the tools is what they need to use to accomplish the task. And then the task is essentially what they need to accomplish. And that's it. And so if you put a couple of those people together or a couple of those agents together, they form a crew that then does one thing specifically. So engineering could be a crew, sales, same concept, they are a crew, marketing, same crew. So what happens usually at the company, engineering starts building the product, for example, when they're done building the product, then they send that material or a bit of that to marketing. Marketing then picks that up and creates marketing material for the products that the engineering team has built. And then sales comes in and sells that product that marketing has kind of polished and created stuff for them to help them actually sell the product. Right. So that cycle continues, but how do you do all of them orchestrate themselves? Right. So there has to be a listener somewhere. Engineering has starting point and then marketing could be a listener, listen for engineering to finish. And then once they're done, start creating the marketing material and then sales listen for marketing to finish and then start selling the product, right? Same thing here with the agents. We have a couple of decorators in CrewAI. We have one that calls start, @start decorator. We have @router, @listen, and @persist. This is very specific and strategic. So @start does exactly what I mentioned. You decorate this to the function or the crew that you want to be the entry point of the agent itself that you've built. Maybe this is where you decorate this, where you pass in the input. The human passes in an input through a function. Maybe there's a— they need to pass in the link and the content type they want to create. So you decorate that function with @start. So that's where it kicks off. And then once that is done, you can add a router basically saying, okay, if this person wants a blog, route it to these specific agents. And so that is a router. It's basically where things come in depending on the input that's coming from the start, route it to this other section where the person wants. And then we have the listener. So most of the crews that you built and put them together, All of those are probably going to have a listen decorator on top of them. So they basically listen for when is their turn to do the work. So the router routes and decides, okay, this needs to go here this time, this needs to go here. And then each of those crews have a listen. If they get called upon, they then execute the task and get things done for the user. So that's, that's the context there. And then persist here, the persist decorator, essentially what it does is persist the state of the run from the top to bottom. So when you start from the top and you run and you can decorate this at the top of the class or in the middle on each function if you want. And so basically if something happens throughout the run, something fails, when you come and restart, it restarts with an ID from that specific failure point and continues from there. So it's like a checkpoint in a sense. So now let's go ahead and jump in and get things rolling, right? So that's already started. Let's go ahead and the first thing here is installing the dependencies. So we need to install CrewAI and CrewAI tools because we're going to need both. And then once that's done, we're going to go ahead and instantiate our API keys. A couple that we're going to use here is OpenAI. You can use whichever one you want. It could be Gemini, it could be any other ones that you're using. Then Serper here, it's going to help us search the web. You can use Exa, you can use Parallel.ai, a couple of them out there. But for this purpose of this demo is what we're going to be using. Then once we're done, we just save that. We add those into an environment variable and let's run that. When this is complete, it's going to trigger that. Now when that's done, the next step is to define the state model for the flow. Now why are we defining the state model? Well, some of these things that are going to be happening within your crew, within your flow, needs to be saved in a state. So that way we remember, okay, this happened here, this happened, and update that as it goes. So for URL, for example, when the user enters the URL that he wants the agents to go do research on and then create a blog post about or a LinkedIn post about, it needs to save that URL in state. So it knows which was used. And also for downstream events that will depend on that URL, it knows which ones to pass when, and it's saved somewhere. Same thing here, content type. When the user passes in the URL, they need to pass in if they want a blog, newsletter, or LinkedIn post, right? So it saves that there as well. So they could be used downstream. And then there's final content. Once a specific crew has finished the task of generating that specific content type, it needs to save that in the state. And this is where we're defining the state, right? And then this metadata, I added that just as arbitrary metadata, if you want to add more stuff in there. So the interesting thing here for you to note is you can customize this base models. You can basically customize all the states that you have in here and add more or delete depending on how you want to do it. But that's the gist of that. Now, why we use state, data persistence, type safety, flexibility, and debugging. Good. Now let's go ahead to creating specialized agents for each content type, which is where we're going now because you still have to create crews for those individual specific content types that we need to create. So first of all, let's start with the blog content creation agent. And I don't want to go into too much detail. You can read through this. But essentially here I have a blog researcher Because it's going to take the link, it's going to do some reading on it, understand the context of that blog or that link that it received. And then one here writes a blog about that specific content that from the— it could be a website, it could be any type of content out there, but you want to write about it, right? And so here it's going to use Surfer.dev to search through that entire documentation, read it. It's going to use Gemini GPT-4 Mini there as the model. And I put max iterations as 5. So it's going to try to scrape that page at least 5 times and then it If it gets 5, it's done. And then we have the blog writer. So the blog researcher does research and then the blog writer, based on the material that he got from the researcher, it's going to write a blog for us. So that's basically the gist of that. Now we go to newsletter. So we have the newsletter one. It also has a newsletter researcher. So you see all of them are going to have researcher. Newsletter researcher does the same thing. Given the link, does research on it, passes it to the newsletter writer that's going to write the newsletter. Good. Now to LinkedIn content creation agent. This one, again, it has a researcher. It will do the research. Based on research, it passes it to LinkedIn writer, which is going to write the LinkedIn post. Good up to that point. Now let's go ahead to defining the tasks. We've defined the agent. We need to give them the tasks. The same here, tasks. We start with the create blog task. Essentially, we have a research task for the research agent and then writing task for the writer agent that's going to write the task itself, that's going to write the blog. That's the gist of that. And I've added extra requirements in there, what it needs to generate, how it needs to be structured and things like that. So we have the description, expected output agent, and then the context. I'm saving context from each one, pass base, essentially passing context from what is done before to the next task. So you can pass context within tasks in CrewAI. So as you go, you would expect it to be, to be much longer, but that's it. Now let's go to newsletter tasks. Same thing here. We have a task for the researcher and task for the writer. Same thing. Same thing for the LinkedIn post here as well. So let's run that. Now we get to the most interesting piece here. Now we need to actually put them in a flow. We have 3 separate crews that we've built, but now is the time to orchestrate them together using Flow. So now we have the content router flow here. And again, just to go over it, we have the persist, basically persists the state across. So the purpose is enable automatic state persistence across workflow executions. And then we have the behavior there. We have the benefits. So you can read through all this. I don't want to read through all of them, but just going to give you the gist of this. And then the start method there decorator marks the workflow entry point. And then we have the router, which given the input, you actually pass in the function that the router needs to listen to and then be able to do the thing. So creates decision points for the workflow routing. And so it routes to either blog, newsletter, or LinkedIn based on what the user has entered. It will route to whichever agent crew that you wanted to actually execute that task. And then you have the listen and it's going to listen for the event. So for example, all of the ones that we have for the blog is going to have a listen with the string blog in it, and then listen with newsletter, listen with LinkedIn for each one of the other ones. So if listen, if it gets mentioned, hey, you got to do the work right now. Let's go ahead and build the class for this, right? So here we import CrewAI Flow. You'll be importing the flow. We have imported the listen router and start. And then we also here importing the persistence from flow persistence here. And then of course the crew, because we got to put them all together. And now here for persist, we're having persist as verbose, basically saying, you know, show us exactly how you're persisting this in what— when you're in your run. And then the content router flow here, we have the class there, and then we have @start. So here's where we're getting the user input. The user enters their target URL, and I have a dummy one there I've entered. And then enter your content type, so blog, newsletter, or LinkedIn post. And then they will enter either blog, newsletter, LinkedIn post. That then gets saved in state, which we had already initialized. So that's going to be saved in the URL. This one is going to be saved in the content type. So that's already saved in state and input is collected once that's done. Now the next step is the router. So this, we just want to route it to whatever type of content the user wants to create. So here we're getting everything from the content from state. So we're getting the content type from state, which is the big determinant factor where we route it to. So we just grab that from that. And we have a listener here that's just going to listen to If a blog is called for them, it triggers this to start running. And basically, here is where we combine all the ingredients to build a crew. So we bring the researcher specifically that we created up there for the blog writer content, and we put them under a crew and we run it. Same thing here for the newsletter. We do the same thing. We listen if it's newsletter, we grab all the agents and the tasks for this specific crew, put them together, and then run it. And so Same here again with the listen one again for the LinkedIn post, similar for that as well. Now, once that's done, we come and we're ready to kick it off and run the process, right? So here we import asyncio here because most of the Jupyter notebooks or Colab notebooks have a lot of processes running under the hood and we're going to manage that so they don't fail. So this is how you do that. And then here we just create the flow itself now. So we have the content router flow from the class up there that we created. And then we have await async kickoff there and we run it and now that's going to trigger to start running. And you'll see it's already asking us to enter the target URL. I'm going to paste the one that I had given it up there, which is a blog from CrewAI. Now let's go ahead and say, let's create a blog, right? Let's go ahead and create a blog. I've given it to create a blog. You can see already the persist true for at least the persist decorator. It's already kind of printing out the verbose statement that we told it to there. It's persisting that to the state and now it's running. So this will take a minute here to run. Once that's done, we'll be able to come here and And print the state so we can see the entire state of what happened. And once that's done, we also have this— I added this extra way of printing out a pretty markdown of the final result. He's going to print out the final blog post here for us to see, to look nice to see and read as well. So that's all it is. We're going to wait for this to finish and I'll jump back and show you the final result. All right. So this is finished. And you can see here, we finished with a whole blog there. But if you look here under the state flow state, this is all the state that's been saved. You can see We have each individual ones. It's pretty long. Now we have to parse through that and just extract just the content itself. And here's what we are doing that. So final content, we are getting final content from the blog there. And here it is, the final content. And you can see here that we have a blog, the future of AI, key players, innovators to watch by 2025, introduction, a thriving landscape. Once you render this markdown, you'll be able to see a good clear content here as well. So that's it. That's essentially what CrewAI flows entail. You could add a plot. And if you plot, it will show you basically the flow of how things ran. It's like a diagram showing you it starts here. This depends on this. This is how it runs. It's a pretty neat way to do that. And then the AND/OR, you can add that as well. And this is for complex logic. So let's say you have 2 things that I have to run before another crew gets triggered, for example, then this is where you would add AND/OR. So if this and this finished, then, you know, start this crew, or if this finishes and this one is not done, you know, kick it off. So you can. Play around with them and you can add as many routers as you want. So essentially this kind of helps, allows you to build a very, very big network. It's like you're going to, you can essentially replicate a company if you want to do with this career flows, because it allows you to orchestrate a bunch of different things with different logics all across. So I hope that was really helpful. Check out these links that I've added on here as well for you if you want to learn a bit more about that as well. Other than that, see you in the next one. Goodbye.