Building a Deep Research AI Multi-Agent with LlamaIndex

Please take it away.

Hi there, Data scamps and data Champs, this is Richie. I am coming to you live from my mum's house in East Yorkshire in England. And please do let us know where you're joining from in the chat, and especially if you're joining from East Yorkshire. We'd love to hear about that. Actually, let me show you one thing cool. My mum says, so if I scoot down like this, looks like I've got a halo, which I'm very impressed with, being feeling angelic today. All right, now, I notice already in the chat, William Long's joining us. William is one of our top question askers. So your task for today, ask more questions than William. There's your challenge for everyone. All right, so with that, let's get started. Now, there are two big AI trends this year. So first of all, AI is getting better at reasoning and that means it's now starting to become useful for research tasks. Secondly, AI is getting better at using tools and that allows it to perform tasks autonomously as an AI agent. And so today we're going to make use of both of these things in order to create a deep research AI agent. And we're going to be using the Llama Index AI development framework for that. Our guest is Laurie Voss, the VP of Developer Relations at Llama Index. Hi, Laurie. Welcome.

Hey, everybody. Wonderful.

So we're basically going straight to the heart of it since Laurie's been working on the Lora Index framework that we're going to be talking about today. Laurie is a web developer turned startup executive turned data and AI evangelist. He's got 25 years of experience in tech and he runs the developer relations team at Llama Index, as we mentioned. So he basically, for his job, he teaches people, people how to build AI applications. Before his time at Llama Index, Laurie was the founding CTO at NPM. He took it from a hobby project to 5 million active users. NPM, for those of you who don't know, is like this huge JavaScript tool, like, incredibly popular. And he also served as a chief Operating officer and Chief Data officer as NPM as well. So with that, please take it away, Laurie.

Thank you so much, Richie. Let's dive straight into our slides, shall we? Or rather our notebook. Today we're going to be building, like Richie said, a deep research AI agent. But to get there, we're going to have to lay down some foundations. Llama Index helps you build agents in a variety of ways. We have a high level system called Agent Workflow, which allows you to get up and running really quickly. We're going to be showing you how to use that and then how to build a multi agent system using that. Then agent workflow is an instance of our lower level workflows classes which we're then going to be dropping down into to show you how to build an agentix system from scratch. And it's using that that we're going to put together our deep research system. So let's go through that. The first thing you're going to need to do obviously is you're going to need to PIP install Llama Index. That can take a little while, so kick that off now while I am explaining the other things that we are doing. The first thing that you need to do is once you've got Llama Index is you need to have an LLM because how are you going to have an agent if you aren't, if you haven't got an LLM model to play with? So in my notebook I'm pulling my OpenAI key out of Google Colab. You can set yours directly and getting an LLM is as simple as just importing the OpenAI class and passing in what model you want plus your API key. In this example, I've been using 4.1 mini because it's nice and new and very capable. You can use your LLM directly. You don't have to plug it into an agent, so you can just ask it to complete any sentence or any question. So in this case I asked it what is the capital of South Dakota, which is my favorite fun training fact. And it correctly tells me that the capital of South Dakota is Pierre. The other tool that we're going to be using today is tavily. Tavily is a tool designed to help agents search the web. It's specifically designed for AI agents, so you'll want to PIP install Tavily Python. This requires an API key as does OpenAI. It's kind of cheating to use Tavily these days or, or kind of unnecessary to use Tavily these days because OpenAI now has web Search built in as a tool that it's a that its models can use. However, I wanted to show how to build a tool in Llama Index. So I'm using tavily as an example of using any third party API in order to be able to show how a tool works. This is what creating a tool looks like in Python. So we've imported the tavily client and it's just four lines of code. We are defining a regular vanilla Python function. This function is called Search Web. It Takes query. We're using type annotations to say that query is a string and we're saying that it returns a string. And we've included doc strings, these built in Python documentation to say that this function is useful for searching the web to answer questions. And then all we actually do in the function is we call the call the tavily client, we instantiate the tavle client rather, and then we call it with our search query and return that as a string. The reason that I've noted all of that stuff is because llamaindex uses all of this metadata as input to the model. This is how the model knows what the tool does. So the model is actually going to be reading the name of the function, so. So you should give it a good name. It's actually going to be reading these type annotations. So that is how it's going to know that there's a parameter called query and that it's a string, and that it should be expecting a string back and it's even going to be reading this docstrings to understand what it is that this tool does. So when you're creating a tool in Python, a function in Python to be used as a tool by llamaindex, it's very important to get all of that metadata as detailed as possible. With a tool in hand, we can now create a single agent using the Agent workflow class. I said that the Agent workflow class is this higher level abstraction that we built to make it easy to put together agents very quickly to get a handle on how they work. And this is what that looks like. So you pull in the agent workflow class and then you call from tools or functions, which takes three arguments. One is an array of tools or functions. In this case, we're just giving it the one search web function that we just defined earlier. We're telling it that our LLM is the LLM we defined so OpenAI and we're giving it a system prompt. You are a helpful assistant that answers questions. If you don't know the answer, you can search the web for information. That's a key part of that prompt is LLMs will often try to answer a question or perform a task by themselves. So when you're putting together an agent, it's often a good idea to mention in your system prompt that you have a tool that will help you with this. So don't be shy about using this tool that we gave you. Running the agent is similarly very simple. You await workflow run and you pass in A user message. Agent workflows always start with a user message. I asked it, what is the the weather in San Francisco? And Tavily came back with the current weather in San Francisco is sunny. Or it was at the time that I was running this notebook. By default, agent workflows are stateless between runs. So every time you run your agent, it's like it's the first time you're running that agent. It doesn't know anything about the previous times. One of the good things about agents is that they can take into take into account previous context. So let's show how that's done. You import the context class and you configure the context to work with our workflow. By doing this, you call context with the workflow class and you get back a context object. Now, instead of just running our workflow, we're going to run our workflow and pass in here that context object that we just created. And I'm going to give it a fact. So I'm going to say, my name is Laurie. Nice to meet you. It returns a response, so far, exactly the same. Because this is, as far as this agent is concerned, the first time I've ever run it. But now if I run it a second time, passing in the same context object, and I ask it, what is my name? It now remembers that my name is Laurie. You can see how you could give this a series of facts or a series of questions and it would remember the things that have happened previously and give you a better experience, That is passing in the context to the agent itself. You can also access your context from within your tools. In this case, I am creating a new Python function to be used as a tool. It's called Set name and it takes context as the first parameter. This is a little bit of Framework magic. If you set context as the first parameter to your tool name. Sorry, to your tool. Excuse me. The framework will automatically pass in the context to your tool. So in this case, we get a variable called state out of the context, we set a key inside of that variable called name, and then we set the state back to the context. So that is what that tool does. Is it given a name, it sets that name into the context. So we create an agent workflow instance here with that function LLM, and you are a helpful assistant that can set a name and we give it an initial state. This is new. This is what that state variable is going to be set to when it begins. It has a key called name and the name is unset. Then, like we did before we configure our context. In this case, it's called the stateful workflow context. And we run it. We run our stateful workflow given that context, and we say, my name is Laurie. This agent has been told that it has a tool and that its job is to set a name it calls that tool. And we can prove that it called that tool because we can access the state objects directly and get the state variable and print it out. And it says that the name is stored in state is lori. So the agent has correctly used the tool to set the name from unset to lorry. One of the, one of the things about agents is that they can take a long time to run. They do a lot of thinking, they do a lot of back and forth. So for a good user experience, you don't usually just want to run a workflow and wait for the answer because the answer could take many minutes. So for a good user experience, you want to stream the output. You want to stream a series of events of how things are going inside the state of your agent so that your user gets a sense that something is happening. This is built into the way that agent workflow works and how workflows work in general. So instead of previously we were running our workflow by calling run and awaiting the response. Instead of awaiting the response, we can just run workflow and what we get back is a handler. Now we can call for event and handler streamevents and it will give us a stream of various types of events about what is happening. In this case we are calling. There are a bunch of built in event types that you can get from this stream. One is called agent stream, which is a stream of text from the LLM itself. You can also, if you want to pull in agent input, agent output, tool call results and tool calls, we'll see those in a bit. But in this case running it gives us a stream of text as the LLM returns it. If I was running this live, you would see each chunk of this text appearing as we go. We can see a little bit more detail by turning off those comments that I had put in to comment stuff out. And you get much noisier output. You can see what the agent input was, that it was a piece of text saying you are a helpful assistant. You can see that it's made a tool call. You can see what arguments it passed to the tool which was current weather in Saskatoon. And you can see the query and the response and the eventual output. Note that the LLM decided what the input to the query should be, we didn't say what is the current weather in Saskatoon? Or rather we said what is the weather in Saskatoon? But the query that it decided to send to Tavle is current weather in Saskatoon. It made a decision about how to use this tool and what the arguments to that tool should be. So. With streaming under our belt, one of the things that you want to do with an agent often is get a human in the loop. Agents are often doing very complicated things. Sometimes they need a human to check their work, sometimes they need confirmation before they make some big action like, you know, sending money or completing a purchase or something. And the framework has a built in way of getting a human in the loop. This is a visualization of what an agent workflow looks like under the hood. We're going to see more of these visualizations later, but you will see that it consists of a series of steps. The blue rectangles connected by events. The events are the green ovals which are used to pass data from step to step. So you'll notice that the events agent, input agent, setup agent, output tool, call tool call result. Those are the things that we were outputting in the stream up here. The events get streamed back to the handler. So to get a human in the loop, we're going to use two new classes, the input required event and the human response event. These are built ins to the framework that you can subclass. By default they take strings, but you can subclass them to take any complicated data structure that you want. In this case, we are setting up a new tool called Dangerous Task, A dangerous task that requires human confirmation. What it does is it calls on the context because we passed in the context here it's going to call waitforevent. So we give it a human response event, which is what it will wait for. And then we give it an input required event, which is the thing that it sends back out to the handler to wait for it. So the input required event takes a prefix which you can set or not. It's just a convenience and we can say are you sure you want to proceed? And we can give it a username so that we know, so that we can check that the event coming back is the same event that we sent out. If you were running multiple of these at the same time. So when the event returns, we take the response that we got and we check if it's yes. If it was yes, then we complete the task. If it's not, we don't. So given that tool, we've set up an Agent. Just like we set up every other agent, we can say give it a dangerous task and you are a helpful assistant that performs a dangerous task. And now let's look at how we actually run this. Just like before, with streaming input, we have set up a handler by calling workflow run and now we're looking for an instance of that input required event. When we see that input required event, we know that the that the workflow is trying to get a human into the loop. So what we do is we call the Python's built in input function which asks for keyboard input. And we are using event prefix, which is that string that we set up earlier over here. So this is the question that it will ask. We don't need to do it that way. We could have set up our input here with the string there, but it was convenient to pass it from the agent. And once we've got our response, we call send event on the context, giving it the human response event that it's waiting for with our response and our username so that it knows who we are. So what happens when we run this? We get our prefix coming out saying, are you sure you want to proceed? This was input via keyboard saying, yes, and the dangerous task is being completed successfully. So now we have reached our first checkpoint. We have completed all of the things you need to do to use an agent workflow, and we've learned some of the tasks about workflows. But agent workflows don't just have to be a single agent, they can be a multi agent system. If you're creating a multi agent system, it's usually because you have several distinct specialities that you want to give your agents. You have. In this case, I've decided that our multi agent system is going to be a research agent that writes a blog post for us. So I have three agents. One is the research agent which will search the web using tavily for information on the topic that I've given it. One is an agent that will write that report and one is a review agent that will read the report and provide feedback. The actual agents that we're going to be instantiating here are going to be very simple. But in a real world case, you can see why you would do this. You'd want to give each of these agents a bunch of specialized logic, perhaps their own workflow, to understand all of the ways that they can research the web, all of the different things that they could do when they're writing, all of the different nuances, when it's doing a review. Our Tools that we're going to use for this, this multi agent system are a web search tool which is the same as the one we've been using the whole time, one that records notes. So when it's completed searching the web for information, it records it to the state. It's going to be a write report tool which actually does writes the report and writes it back to the state so that it can be reviewed and then a review report tool that actually does the review and provides feedback. So here are these, here are those tools again, they're just vanilla Python functions. The search web one we've seen before, the Record notes one, is very similar to the set name one that we used earlier. It gets the state, it sets research notes into the state and then it says that that was recorded. The writerport function is given the report written by the LLM and it just writes it to the state as well. And the review report is given the review and it writes the review to the state. So these tools are all doing very similar things. With those tools defined, we can now create our agents. So there are two types of agent that you can use when you are creating an agent to be used in a multi agent system. With agent workflow, one is a function agent and one is a react agent. The function agent is the better one. It is able to use tools natively and gets much better results. Not every it uses tool calling under the hood, which not every LLM model supports. So if you can't get function calling agent to work, then you can fall back to react agent which works with every LLM. But in this case we're using OpenAI which definitely supports function calling. So we can use three different function agents. So I've set up our research agent. I gave it the search web tool and the record notes tool and I gave it a prompt. Here's the right agent with a prompt about how it should work. And here is the function agent with a prompt about how it should work, matching those descriptions that I just gave you. With our agents defined, we can now create our multi agent agent workflow and run it. Previously we were taking agent workflow and calling the from tools or from the from functions or tools. And this time we are calling agent workflow directly and passing in a set of parameters. So in this case we're setting in sending in an array of agents. So it's the research agent, the right agent and the review agent. We're setting a root agent which is which agent is in charge when the multi agent system starts to run and we're passing in an initial value of our state. So we did this before. You can see research notes, you can see the report content, and you can see the review content both set to something indicating that they are not done yet. How this works is that the agent workflow will pass control back and forth between these agents. Each agent will do whatever task has been set to it and they have been instructed in the system prompt that if they run into a task that they don't know how to do that they should pass off to some other agent. You can see us sending hints of that nature using this parameter, which is optional. So I've said that the research agent can probably hand off to the right agent, the right agent can probably hand off to the review agent or back to the research agent to do more research. And the review agent can pass off to the research agent and the right agent. You don't have to pass those in. It will try and figure it out itself. But it can be helpful to put in some guardrails to say what should my agent, what other agents should my agent know about and be able to pass to? So we can now run our multi agent system just like before. We're setting up a handler by calling agentworkflow run and we pass in a starting user message. I've asked it to write me a history, a report on the history of the web. This is exactly the same as our previous event stream handling. We've just put in a bunch of basically pretty printing to give us a nicer set of log output when the agent is running. So when it sees an agent name, it will print out the agent name. If it sees agent output, it will print out the output nicely and it will show what tools it's planning to use. Tool call results and tool calls are also going to be handled. So when we run that, this is what the output looks like. You can see a very pretty agent. Research agent starts off, it searches the web, it records some notes. It does that multiple times because it searched the web multiple times about different things. And then it has decided that it is done doing research. So it calls a special tool that we didn't have to pass in. It's built in, called handoff. And this handoff tool passes it off to a different agent. In this case it has picked the right agent, that is the writer agent. The writer agent takes the stuff that is in the state, the research notes that has been set up earlier, and it writes a report and then it calls handoff to the review agent. And the review agent decides that Everything is cool with this report and I approve the report as it stands. So this is what it is actually spit out. It is spit out some nicely formatted markdown with a reasonable summary of what has happened to the web since it was born. We can get the report itself because it's in the state as report contents and we can also get the review from the state. So this is our second checkpoint. We're doing pretty well for time. We've now successfully put together a multi agent system using agent workflow. So agent workflow is designed to be simple, easy jumping off point for putting agents together. So you can either just use one agent or you can put together a simple multi agent system where they can pass tasks back and forth to each other. But there are some things that agent workflow can't handle and they revolve mainly around fine grained control. Agents are autonomous. They can make decisions for themselves. That is both a strength and a weakness. Sometimes you know exactly what sequence of things you want your agent to do, and sometimes you want your agent to decide for itself what is the right thing to do. So there's a combination there of structure and autonomy. You want to be able to say, do these three steps and then you get a choice. Do these three steps and then you get a different choice. You want to be able to pass that stuff back and forth. Agent workflow is also designed to just pass in and out strings, but you might want to pass in complex structured inputs and get structured outputs back. So you know a complex JSON object in and a JSON object back out. If you were putting this, if you were putting your agent under the hood into a larger piece of software, if you wanted to handle images as well as text, if you wanted to do query planning, if you wanted to do reflection, Query planning is when you get your initial question, your initial user message. You can get the LLM to look at it and decide what is the best way to tackle this question. Reflection is sort of what the review agent was doing just now. You can get your agent to look at its own output and decide whether it was good enough and if it was good enough. If it wasn't good enough, it can go back and try again. We're going to see exactly that technique used later. So to do this, we're going to create our own workflow. Workflows are regular Python classes. Agent workflow is just an instance of one of them, which we wrote for you. So let's pull in some new inputs, some new imports, a start event, a stop event, a Workflow and a step. And let's define a new class called my workflow, which descends from the workflow class. The way that you define the steps is you use this, which is called a decorator, step decorator. And that is, and we are defining a Python function that takes a start event and emits a stop event. And the stop event has the result. Hello world. This is the simplest possible workflow. It has exactly one step. It is very trivial. It is just to show you these components of a workflow. Workflows are classes, they have steps. The steps are defined by the step decorator. And we heavily use class annotations or yeah, type annotations to indicate how they work. So the step that accepts the start event will always run first in a workflow. And any step that emits a stop event will stop the workflow. So the way that you run a workflow is exactly the same as we ran agent workflow, because agent workflow is a workflow. So we create our workflow with a timeout and we call workflow run that we await and we get our result, which is hello world. This is how you run this stuff in. So far we've been showing you how to run workflows in a notebook. Workflows are async, so if you're running them in vanilla Python, you need to do one extra step to make sure that they can run, which is you need to put them in an async function like this. This is not necessary in a notebook, it's just if you're running it in vanilla Python functions. I mentioned earlier that you can get that you're going to see visualizations of workflows. This is a built in feature of the framework. It will generate a picture of what your workflow looks like. To do it, you install the llamaindex utils workflow class and you call the function draw all possible flows. What this does is it will spit out an HTML file, actually. And when you open the HTML file, you get an interactive visualization of your class. So in this case we are seeing the visualization of the simplest possible class, which is a start event. Going to my step. Going to a stop event. I see a question from William. What level of human intervention can be provided I supplying data or reference documents or an interpretation of that data or correcting one of the misinformation hallucinations? Yes, you can do all of those things for. You're asking about human in the loop. Human in the loop can provide any level of input. If you subclass those input Required events, and those human response events, you can pass in fully complete feedback, not just strings, but full objects of feedback or PBS or whatever you want, back to our workflow. If you want to do a workflow that is more than just one step, you need to create the events that pass the data back and forth between steps. You can see us doing that here. They descend from the event class. We've created a first event and a second event. Now we can define a workflow that has three sets. Step one accepts the start event and emits a first event. Step two accepts that first event and emits a second event. And step three accepts the second event and emits the stop event. This is very similar to the one we were just looking at. And you run it just like you did before. And this is what the visualization of that chain looks like. So you can see step one is passing a first event to step two, which is passing a second event to step three, which is doing a stop event. So this is a chain that we've implemented here. Obviously this is not a very interesting workflow. This is just a linear workflow that doesn't do anything in particular. Let's see what happens when we add actual logic to these things. The first thing that you might want to do is you might want to do a loop. We talked about reflection. A loop is how you do the reflection, which is you need to be able to go back to an earlier step. So in this case we have the same step one as before. It emits a first event, which goes to step two. And now step two is going to make a choice in a real workflow. Obviously it would be an LLM making this choice for some on based on some input. But in this case we're just doing random math. So depending on whether it gets a zero or a one, it will decide a bad thing happened or a good thing happened. If a bad thing happened, it will return a loop event. And if a good thing happened, then it will return a second event. So you can see in our type annotations that we have said that step one accepts either a start event or a loop event. This is how workflows knows what to do when an event is emitted. So if a loop event is emitted, then that will trigger step one. Again, if a second event is emitted, that will trigger step three. Because step three in its type annotation says I accept second events. So it's not the order in the code that defines how things happen, it is the type annotations and what events get emitted. So you can see in this run, it started the workflow, the first step completed, the bad thing happened, it went back to step one, the first step completed again. Then a good thing happened, the second step was complete, and then the whole workflow was done. That's what this looks like when you visualize it. You can see step one goes to a first event to step two, which can do a loop event back to step one, or it can do a second event back to step three. Then it gets a stop event, and it's done. The same logic that allows us to do loops like that allows us to create branches. So a branch is obviously just an if statement that your LLM can decide it wants to go one way or another based on a. Based on a decision that the agent makes. So to implement an example of that, I've created some more custom events. A branch A1 and a branch A2, a branch B1 and a branch B2 event. And again, we're making a random decision in our start event whether we're going to go to branch A or whether we're going to go to branch B. If we go to branch a, we emit a one event, which goes here. A one event emits an a two event, which goes here. A two event emits a stock event. Or branch B1 will go to step B1, which will emit a branch B2 which will go to step B2. That's what this looks like when it visualizes. You can see the start event is emitting either one or the other of these, and then it's just a chain after that going to a stop event. There is more to. There's more to what workflows can do than just branching and looping, though. You can set up concurrent execution. This is very useful if you have multiple tasks that you want to complete in parallel. For instance, if you have a complicated piece of work, such as writing a report, for instance, that you could split up into multiple questions, you could split up each of the questions and run them all simultaneously. Or in a guardrails situation, you could take the same query and pass it to two agents simultaneously. One of them would try to answer the question, and one of them would check whether or not the question was an acceptable question. You could check whether the user is asking to do something illegal or trying to do a jailbreak or something like that. And you can run those two things in parallel. So if the guardrails agent says, oh, this person is asking to do something illegal, it can prevent the output from the other agent being output. So here's what that looks like again. We have a regular. We have set up an event custom event class, We've created a workflow class. We've set up a step. And the only thing that we're doing differently is that we have given ourselves a context object. This is the same context object that we were talking about when we were talking about agent workflow. And on that object we are calling send event and we are emitting three events, one after the other. So instead of returning an event, which allows us to only emit one event from our step, we're calling send event, which lets us emit as many steps as we want. And then in step two, it gets step two is triggered by any step two event. So all of all three of these step events will trigger step two. You can also see that we've set the concurrency here by passing a parameter to the step decorator. In this case, it can run up to four times at once. So we instantiate our parallel flow. We call run on it. And you can see that it is running each of these queries running slow. Query 1, Query 2, Query 3. But a thing that happens here is the output. It returns a stop event, and we output the stop event, and we can see that the output is only query one. That is because the first stop event to trigger triggers the end of the entire workflow. So query 2 and query 3 never complete because query 1 finished first. Sometimes that would be useful, but more often you would want the output of all three of your events, which means that you should collect your events. So here's how we do that. This step is the same. We're sending send event three times. Step two is the same. It is running the query. And it's. But a change is that we are emitting a step three event with the output of that run. And in step three, we're calling a new function we haven't seen before on the context, called Collect events. So Collect events takes the event that you're looking at, and this argument is what events it should be looking for. So in this case, we've said you're looking for three different step three events. How this works is Collect events takes events that it receives and it stuffs them into the context. If it has received all of the events we've told it to wait for, then it returns those events. If not, it returns none. So you would expect it to return none most of the time, because it's going to trigger multiple times. Every time a step three event event is fired, this function is going to run and it's going to return none. So if we get none from collect events, we just don't do anything. If we do get something, we print the result. So what that output looks like is exactly the same as before. We are running query one, query two, query three, but we see it run three more. We see step three triggers three times. The first two times it says not all events received yet because it's only received one event and then it's only received two events. And then the third time it receives all three events and so it outputs those three events. It's also worth noting that because these things are concurrent, the events don't necessarily return in the order that you sent them. So in this case, we triggered queries 1, 2 and 3, but we got them back in the order query 2, query 1 and then query 3. In this example, we've been collect where we've said we want to collect three different step event, step three events, but you might not necessarily want to collect events all of the same type. So I've created a more complicated workflow class here which has three different chains. A step A event, step B event, and a step C event. And we have emitted, instead of three of the same event, we have emitted three different events. A step A event, a step B event, and a step C event, each of which has its own step. These steps all emit a type of complete event. And we've said that step three in this case should accept all three of those types of events. So no matter which one of these functions returns, it will trigger step three. And we've said in our collect events, we've given it an array of events and said wait for a step C, a step A and a step B. So now you can see that running. We see it doing something aish. Something B ish, something C ish. We can see it receiving events query 1, query 2, query 3 and then it prints out the events it received. An important feature here is that the order here is preserved. So we said when we were collecting events that we should that we are waiting for a step C, a step A and then a step B. And that is the order in which we got them back. We got a step C, a step A and step B. When you visualize this flow, it looks quite pretty, which is why I include this visualization. You can see us with our start event and emits a step, a step B, step C, which triggers those steps, which triggers those complete events which all go back to step three before they get completed. Just like Our other earlier steps workflows, you can do streaming output. The agent workflow streams a bunch of events automatically for you. When you are writing your own workflow, you need to stream those events yourself. The way that you do that is you call another method on the context called write event to stream. So you have to explicitly say which events you want sent out to the handler in this chain. In this workflow. In step two, I've created a generator that is using the async version of our streaming completion from our LLM. Every time this generator generates a chunk, I am going to write an event to the stream called a text event containing the chunk of that text. You can see us handling that output here. So just as we have previously, we instantiate a workflow. We call run on that workflow. We don't call await because we want a handler. And then we look for the handler's stream events. If we see a progress event, which you can see us sending here, we write event to stream a progress event saying step one is happening, then we output those. If we see a text event, we output those as well. When we run it, you can see step one is happening, then you can see all of the output from the LLM, then you can see step three is happening, then the final result. Cool. We have reached another checkpoint. You've now put together a workflow from scratch and you've learned the primitives necessary to build any workflow. So. At this point you could put together a multi agent system of your own, just by having multiple steps with agents in them which talk to each other. Let's see if we can, in the time that we've got remaining, actually do that. So this is going to be our deep research multi agent system. It's going to have three agents. One is a question agent, one is an answer agent, and one is a report agent. So the question agent is going to accept a research topic and it's going to generate a whole bunch of questions that are necessary to write a report about that question. An answer agent which is going to take each question and answer it. So we're going to need to call it many, many times in parallel, and then a report agent which is going to take all of those answers. Oops, it's going to take all of those answers and generate a report. So we create this as a series of agents, just like we did before. We create a function agent with a system prompt. One for the question agent, one for the answer agent, one for the report agent. We give them all the same LLM, the Function agent is going to have the search web tool that we created earlier because. Sorry, the answer agent is going to have the search web tool that we created earlier because it's going to be the one that actually searches the web to try and get the answers to the questions. And we are going to create quite a complex workflow here. So it has generate events, question events and answer events, as well as progress events so that we can stream things back to the user. We have a setup step which emits a generate event and then the generate event, generate event triggers generate questions, which is where we call the LLM for the first time, or rather we call our agent for the first time and the agent calls the LLM. We say generate some questions on the topic that we received from the user. So we await the response from the agent. In this case, it's going to be a series of questions, one question per line separated by new lines. And we use some basic string manipulation here to split those up into a series of a series, an array of strings. We record how many of those questions we got because we're going to need to wait for that, any questions, and then we fire off send event for each of those questions. So for each of the questions that we got, we send a question event. That means that we're going to trigger. If there were 20 questions, we're going to trigger answer question 20 times. Answer question, uses the answer agent, gives it a prompt, gives it the question, waits for the response and emits an answer event. And here in Write Report, we see the collect events method that we were using before. We wait for answer events and the number of answer events we wait for is the total number of questions that we set into the context earlier. If research is, if it hasn't, if research is none, that means that it hasn't yet received all of the events that it's waiting for. So we send a progress event saying that it's still collecting answers if it's done. We say generating report. Then we pass all of the questions and answers to our third agent and say, hey, generate a full report. You can now see us running this. We're calling workflow run and we're giving it a research topic, a history of San Francisco. And we're passing in those three agents that we created earlier and in our stream events. We are outputting progress events as we go and then we're going to output our final result. So you can see a huge amount of output happens here. But basically this is the questions and answers happening as they go. So let Me scroll up to the top of the input the output rather so you can see, it starts research. The research topic received is the history of San Francisco. And then it receives questions and comes up with answers one after the other. So it's collecting answers, collecting answers, it's collecting answers. All of that happens until finally it decides that it has received enough and it generates the report. And our report is some very beautiful markdown containing an in depth report on the history of San Francisco based on all of the answers that it just got. So that is a multi agent system built from scratch using Llama Index. If you've got any questions, now is a great time to ask them. If not, there's one bonus section that we can go through that'll take us one minute more in which we add reflection. But Richie is here to tell me we don't have time for that. All right.

Might have to leave that one as homework because we've only got a few minutes left and I've got some questions for you.

Okay.

And I think the audience has a couple as well. I'd say that was really great stuff and I very much enjoyed it. I thought we were going to be attacked by a pack of wild dogs halfway through, but we survived.

I was hoping my noise cancellation would cancel on my dog freaking out, but I apologize.

No problem. All right, so one thing I'm curious. So it seems like the multi agent setup, you had a researcher agent, you had a reviewer agent, you had a reporter agent. Is that a common setup or are the different ways, like how do you divide the tasks into different agents?

Well, this is an example. So these agents are a little bit simpler than they would usually be, but essentially a workflow creates an agent. So you can imagine that each of those agents that I created was its own workflow with its own human in the loop and doing stuff in the background and things like that. The rule of thumb of when you create your own agent to do something is when it's got a task complicated enough that it has specialized logic that you want to implement that doesn't apply to other parts of the workflow.

Okay, I like that. So just if it's, if it's kind of a discrete task that it wants to do, you want separate agent for that rather than, I guess, like there should be clear responsibilities, I guess, like real employees.

Is that right? Yeah, it's. It's when they need to be. It's when they are comp. They have a task that is specialized enough and complicated enough that you would want custom logic that you wouldn't want to repeat in other parts of your app.

Okay, nice. The other thing is. So you made use of the idea of, like, a parallel workflow. Now, anytime parallel code happens, usually disasters happen or there's something complicated. So how much thinking you have to do, like, what do you need to do to make sure all the parallelization works?

Well, most of the pitfalls are handled for you by the framework, because that is. That's what the framework is for. The key thing, which I think I called out when I was talking about it, is you need to remember that the order of the events that you get back is not guaranteed. So if you fire off 10 events, you can't be guaranteed that the 10th event you get is exactly is the answer to the tenth question you sent out. You need to make sure that you are preserving enough state in your questions and answers or whatever task it is that you've received that you know that. What is this the output of?

Okay. All right. So just as long as you don't care about, like, what the order is, they're sort of independent things, I guess. Anytime you want a for loop or something, or like. Oh, never mind.

Just.

Yeah, anytime you do, like, you don't care about the results, it's probably okay to do things in parallel then. Is that about right?

Even if you do care about the results, it's possible to just, you know, make sure that your. That your returning results stay what they were related to so that you can put them back together into the order in which they were set. Okay.

All right. The other thing is, so at the start, when you were doing the simple example of, like, my name's Laurie, then what's my name? It seemed like it was automatically remembering the conversation. So I'm. Do you ever have to manually care about, like, what gets remembered and what doesn't? In fact, I think strong. In fact, there was an audience question related to this. Let me put this on screen. So this was from William saying, is there a way to control the preset states and context? So, yeah. What happens if you want it to remember some things but not others? How do you manage that?

That's an excellent question. The agent workflow will, by default, remember everything for you. That is sort of the agent workflow all over is that it's doing a convenient basic thing on your behalf. If you want to be specific about what things get remembered into the context and what things don't get remembered, then you want to build your own workflow, and you can manage what gets set by using the Context variables, you can just say this gets remembered. And when you are passing stuff to your agents or to your LLM, you pull stuff out of the context and say, here's the specific stuff that I wanted remembered.

Okay, all right. So you do have that fine grain control when you want, but by default it's just like everything's handled for you. You don't need to worry so much.

Okay, yes, by default it just tries to remember everything.

All right, one more question for me before we wrap up. So you mentioned there's a function agent and there's a React agent, which is. So the function agent's the sort of the one you want to use first. But can you clarify what the Difference between the two is?

React is a pattern of interacting with LLMs which existed prior to model creators handling tool use natively. So in a React pattern it is literally just the LLM says I am going to use a tool now and that is enough for the code to actually decide to call a tool. It's a very basic pattern and it's a little bit flaky.

Okay, the first sounded good. It's like basic, very nice. I like that. Then flaky, not so good.

Yeah. So the model, because React was how tool use was invented, but it was a little bit flaky. The model creators created native tool use and that is what function calling Agent does and that is much more reliable.

Okay, all right, nice. Okay, so we are coming up to time. Oh, there's one more question from, from Dietrich that we ought to cover, so. Very interesting, go along. Is there an introductory or advanced course about the subject available in the members area? So yes, we do have some courses on agent, but I can't remember what the title is off the top of my head. There are tons of things coming on the way. So you, what you really want is probably developing LLM applications with LangChain. Actually there's a whole series of courses on developing LLM applications. Oh wait, we've got Llama Index thing on the way. Which would be more appropriate given the session.

Thank goodness.

Yeah. Okay. Search for AI agent in the bar. That's better.

If they're looking for in depth stuff, the Llama Index documentation has lots and lots of resources about building, about going deeper into building agents With Llama Index.

Absolutely.

Yeah.

Basically there is an AI developer skill track. So that's the thing you want and it starts from scratch. So yeah, plenty of courses.

Michael wants to know if this has to run in notebook only and. No, this can run in vanilla Python. People have this in production as actual Python APIs.

All right, nice. Okay, so before you all dash off tomorrow, we've got a session on how to build your career in finance using Python. On Thursday, there's a session on how you run an AI transformation program, so slightly more management oriented. And then next week, we've got a session on. On the Thursday on enabling generative. Generative AI in the telecoms industry. So lots of great sessions coming up. Thank you once again, Laurie, for that session. Very, very informative. I very much enjoyed that. Thank you to everyone in the audience who asked a question. Thank you to everyone who showed up today. I will see you all again soon. All right, cheers. Bye.

Thanks, everybody.