Live on TwoSetAI: Stop Your Agents From Spinning

I don't know. Can you hear me?

I do.

Yes, but you can hear me. But let me see if I can, um, let me see. Can you say something? Sure. Okay, you know what? This works. I just cannot hear you. But I have to hear it on my phone. This is so strange.

You hear me on your phone but not directly here?

Oh, I hear myself. Okay, all right, the first conversation. Sure. Hi. Hi. Okay, you would work. I just cannot hear you. Could I help? So this is going to be so funny. I'm sorry, everything's delayed. This is what I'm going to do. I have a few slides. I'll just put them first and then I'll just hand it over to you and then we can wrap up. So it's a little weird, but let's figure this thing out. So everyone, if you're here present, I'm so sorry. In my studio, I cannot hear Nicole, so I have to we have to have a workaround, but please bear with us. This is just— it's weird, isn't it? But let me first remove— okay, adding myself back. Something will work out as long as the audience can hear you, you know. This is called the curse of a live demo, isn't it?

Yes.

All right, okay, um, thanks for everyone joining today. Um, hello everyone, I am Angelina Yang Yang. I run I'll go to our second slide. I run TwoSet.ai podcast, and initially the idea is just two person talking about AI. And when I started this, I was thinking, you know what, instead of showcasing prototype ideas, what if I just ask the experts, the proven founders, the builders like Nicole here today with us to share what they're seeing actually in production? And so I spent last year interviewing more than 100 AI founders and domain experts. And I have received thousands of questions from the audience. Here are some of my past guests. If you're interested in checking what insights that they shared, do tune in on the channel. And I've decided that I'm going to set up a series of workshops that I can get these experts, builders, you know, Nicole here today, directly in front of you, sharing their insights of real production pain points and answering questions on the spot. Nicole was on the show and our episode will be tested, will be released soon, and I'll share with everyone here. I'll give today, she's here today with us and she will share some ideas of one of the biggest pain points of building AI agents. I mean, I can't speak more highly of our guest today. She's a dear friend of mine. She's written two books with O'Reilly. One of them is called The Definitive Guide for Building AI Agents. And she's also teaching at different venues, including Oxford, O'Reilly, and well, now with TwoSet.ai as well. So today's talk will be around 30 minutes, and we will have time for Q&A. Thanks for everyone who's joining today. And sorry for the hiccup at the beginning. Before we get started, please do share in the chat, like, what's your name, what you're building, what you're working on, and where are you located? Let's get started. I will hand this over to Nicole. And Nicole, I'll mute myself and I'll listen to you on my livestream. I'll see you there.

Yeah, you do that. I hope everyone can hear me. All right, let me see. I hope everyone can hear me right now. I'm just seeing the private chat. Do I see any comments? Do I see anything else? Let me know, guys, if you hear me or Angelina or someone since we are testing this. Maybe Angelina, if you can hear me there, just send me a short— Nicole, I'm so sorry to interrupt you.

Do you have a slide that you want share, you can— you have to, you know, click on, uh, so everybody can see.

I know, I was just wanting to make sure that everyone can hear me. All right, so in any case, I hope people can hear me, and then I'm just sharing my screen. So that's the first one. Angelina said we have some hiccups. I hope all is resolved. You can hear me. I'm not talking to myself alone and presenting for myself alone. So yeah, today's topic is about stop your agents from spinning, and that is something which cost me a lot of pain points, and that's really a problem I looked into. And let me just move on. So a little bit about why this is important. And actually one thing, Angelina, if you hear me, I still hear some background noise while I'm speaking, which is kind of annoying. And interrupting. Because that makes it very hard to present. Alright, so now that is gone. Perfect. Now I hope I can present and everyone hears me. All right. Perfect. All right. So a little bit about the background, why stopping your agents from spinning is also important and give you a bit of a deeper understanding. So a little bit back during the machine learning times and also earlier stages with Transformers and large language models in general, we had kind of that notion that bigger, better, and more equals kind of better systems, which is actually not the case. So that's, this was really for that long time, that kind of a dominant AI pattern. So you had a task model, and then your output. So yeah, everyone kind of arrived at that assumption. So if a performance is poor, the fix you have to go for is, alright, you choose a bigger model, different model, better prompts, in terms of large language models. And now what we arrived at is that kind of, all right, so if we have more agents, then we certainly get a better system. Well, but that's a problem. That's actually not the case. And the thing is that if you're having multi-agent system, that really change your whole equation. So the poor or the sad thing or hard thing, the hard lesson I learned is really that Yeah, model size or agent count really doesn't fix your system failure. It's even worse, and I show you, I brought the receipts. So yeah, in general, your problem shifts from general what you asked before, can the model solve this task, to something more closer to can actually your system execute really that stable sequence of decisions without making any failures, without introducing any failures, without spinning, without interrupting, without causing any problems or errors. So you see a short illustration of that agentic system paradigm. So you have your task planning, you have different valuable, available options, you have tool interactions, you have different states, what you are tracking. And then you have, of course, your coordination and your messaging between your components between your agents. And then eventually, you will get a final output, hopefully, if a task is solved. And another thing also is what I want to remind is that communication is not equal coordination. These are really two different things. They need to be treated as two different things. So communication is, is what happens in terms of message exchange between your agents, information passing. And the coordination is, is really that strategic control of activity between the agents. So for instance, supervisor or lead agent spinning up another agent to break down a task and fulfill that task and so on. So that we have different paradigms. And the problem also is that one size in general doesn't fit all. So we end up or ended up with, with of course multi-agent system with different variants with different architectures. And they are also defined then by their structural topology. And I wrote an article for O'Reilly. It's really available. So if you are interested in this, slides are shared with you. That's the link to the articles and how to reason about how to choose actually the different architectures. So if you want a guideline or a rule of thumb how to choose your architectures, that certainly helps you. So basically what we are trying to achieve is, all right, so you have your task type, you, you have your— to choose your topology, then you choose your communication protocol according to what you're trying to achieve. And there are different combinations, and oftentimes you have some hybrid architectures, and then you get your system output. And also, your overall multi-agent performance depends actually on your topology you're choosing. So you have those different structural architectures. So you can have that centralized multi-agent system, what I mentioned, with that architect— with that orchestrator, with your supervisor. And then you have your subagents or your lead agent. And then you get your subagents in doing the task. Or you have it decentralized. So you have different agents. Where they are working together, or you're having independent ones. There you sprawl them out. And then you get an aggregation, so swarms and things like this. Or you have then the hybrid ones. There you might have in between an orchestrator, subagents. You spin up whole swarms of agents doing some research, bringing feedback back, aggregate it, put it to the orchestrator. The orchestrator— delegates it to another sub-agent to do some analysis with it, and then it goes on. And also, each topology, of course, distributes communication, the coordination differently. And all of that directly really affects your system's overhead. It accounts for the latency in your system. And most importantly, that matters for your failure propagation. And also, none of all those different architectures you're finding out there, all those different topologies, they will not eliminate communication. This is why I brought it up. That's an important part. And they also don't move any coordination. So they are just relocating it to different bottlenecks and to different failure points. So if we have any systems engineers in the audience, they might have seen that law already in during their education during class. And the formula, or what you're seeing there, to some it might look scary, to some it looks very simplistic. However, that, that is the product reliability rule. And this is just applied to multi-agent systems. And it's well known in systems engineering. Luster has written this in 1960s somewhat. So it's really, really old. It's nothing new. It was obviously before multi-agent system existed. But it is tied to how different components tied within the system are working together. And if they are executing sequences or executing things in sequence. And what he found out was that the overall system success actually depends on the product of the individual components of its success probabilities. And this is what the math formula really represents. So this represents If you are having sequential components within your multi-agent systems, if they are working together, then the overall success will depend on that formula. And this brings me directly to that sad production failure pattern. So if you have agents which are chained, outputs become downstream inputs, of course. Then any small schema drift becomes your cascading failure. And now you might think, all right, so, but yeah, I have that structured outputs via the API and I'm good, all is fine. But well, what about if you're calling any API in between, if you are processing the data? So that leaves then the realm of that structured output from the model provider. And also, there might be some failure points with that, although oftentimes it's fairly reliable. But there might be a slip in some cases. So you need to be sure because if you are not, the failure will definitely propagate through your system. So speaking of which, so this is why here on the slide you're seeing that the priddle, chase, and trap. So, and why I recommend or say or advocate for using validation. And that's really important part in breaking that reliability rule. So, and this is a concrete example, and I will show you a notebook where you can actually find and do the computation yourself. But just bear with me. So if we look at agent, at just one agent, and we assume, which we all know it's not the case, if we assume an accuracy of your LLM, of your agent in doing just the task, right? And we assume it's 98%. So it's 98% accurate. So you have only that error rate of 2%, right? So if you go to 5 agents in that chain, that already drastically changes. So you are nearly at 10% error rate. And if you have even more agents, like 10, that drifts even more up in the error. So you are around 18%. And as, I mean, as you see here on the slide, this can get really nasty, right? So if you are not controlling the output as the input to the next agent, you're just propagating the errors through your state, through your whole system. And thus, yeah, it makes the system unreliable. And this directly leads to that bad cost of agentic failures. And I don't know if someone of you guys actually experimented with OpenClaw at the beginning. But I didn't do anything recently. But I know at the beginning, there was just like, it was just that autonomous part was just blindly retree and retree and retree and not optimizing things, and I will get on that point later and how the math works out there too. But the problem was then, well, in 5 minutes that thing just spent for some, some very, very simple task, just $30, just, just in 5 minutes, it just burned it. And this is what happens in your production agents as well. If you are not controlling the TreeTree, if you're not controlling the validation, if you're not controlling your agents properly, they will just burn your tokens, they will burn your credit card, your money, and they won't succeed with the task at hand. So that's really the problem. And what happens then is latency spikes, you have latency amplification, of course, token cost explodes, you have a lot of re-tree loops, and that can get really bad. Then tool call cascades and you get non-reproducible failures because it's just acting weird, your system. And this is what I meant then when I said I show you the retrace, how this works out in your systems. So yeah, how do the retrace actually compound? And this is just a simple math example. So to really get a high low overall success rate and you have weak components in your system, weak agents, they may need many retries. So just as an example, I mentioned with OpenCLAW. So, and this can lead to a lot of retries. So here you see the base step success, the number of retries, and then your effective success rate after that retry. So given that, if your base success is 0.3 and you want to achieve that 98% I mentioned earlier of effective success, you actually need 11 attempts to do that. Now, if you think about that, so if you have 10 agents in your system and your base success is only 0.3%, and you have 11 attempts for each agent. So this is 11 times 10. And now you can see where I'm going. So this is a lot of latency and a lot of tokens and a lot of overhead you're adding. And actually your clients won't be happy about this as well. So what happens is that's the hidden cost. So you have a low base reliability, you have uncontrolled retry loops, then you end up with higher compute costs. And most importantly, and that's the problem if you want to build a successful business around multi-agent systems, you, you want to look at your tail latency. And this is your hidden costs eventually about that. So as a hidden cost on all those problems, when your agents start spinning is that you have compute, you have the money you're throwing in for your APIs and also uncontrolled model calls, MCP calls, things that just don't go right because there's some weird input to that, they get a weird output and so on. And then you increase your tail latency. And what happens is then if you are actually doing that brute force, then your long tail latency will increase. So if you don't know, in production, if we have agent systems and also LLMs in general, your user's experience is really defined by the tail latency. So this is the P95 or P99, depending on how you're looking at it. But basically, this means 95% or 99% of your users get this speed. But if we go with 95% of your users get a good speed, a system will still feel bad and broken for 1 in 20 of your users. If you have thousands of users, that's a lot of users not being happy. And if we all know, the bad words spread faster and further than someone being happy. So you obviously, you want to avoid that people get stuck and they are unhappy and you have latency spikes and all of that. So you want to aim at 99%, getting really good speed and everything you can avoid or do to avoid unhappy customers and unhappy users. That's what you want to do. So this is why it's important to have that, our address failure propagation. So this is just an example with and without validation gates. And here with validation gates, it's super simple. I'm just using Pydentic models and use that to break the multiplicative failure chains into local recoverable events in my multi-agent system. So you see the smaller graph is what happens without validation. So we have output 1, then agent 2 consumes it, agent 3 consumes it, agent 4, and so on. And this is what I showed you with the systems failure, with the loss of SLOR slide. Well, this adds then up to that 18.3% or 9.6% if you're looking at just 5 agents. And I mean, 5 agents, it's not that much, right? So even with 3, it was about 5%, I think. So yeah, it still adds a lot. It adds a lot of latency. And then the larger plot, what you're seeing here is if you add validation gates. So you have that gate. And if it's invalid, you retry or re-repair. And so the agent output from agent 1 gets over the gate. If it's valid, it's allowed to go to the next agent. The next agent is allowed to consume. There's another gate. If that is valid, it goes to the next agent. The agent is allowed to consume. And then what you get is you get your stable propagation. And you don't propagate your failures through the system. And if you do not get this, then you just retry and you repair. And there's actually a very sophisticated and very easy way of actually doing a proper retry and repair what your system is doing or your agents are doing. And that brings me— and the title is by choice. Meant to sound a bit opinionated and sarcastic. So it's really from that stochastic hope that eventually, if you do enough retries, sometimes, somehow you will succeed, goes more, you're really doing this via proper engineering. So you get to the more deterministic engineering. This is why I choose that title. So you're enforcing the generation with the schema enforcement. And I do this with Pydentic. And what's happened is then you will end up with your proper error propagation for the system. It won't be that it goes all crazy, but you can actually stabilize. You get about 2% instead of 18% error rate. So this is estimates and this depends of course on the system and it assumes independent events, which is true with agents. But yeah, I hope you get the gist of what I'm trying to tell you. So if you don't interrupt the failure propagation, if you're not properly validating the inputs and the outputs and then the inputs to the next agent, this can go very fast. Wrong very fast. And here you see the error propagation in multi-agent system to give you a bit of a better understanding. So you have your root user request, agent 1, then agent 2. All right, so the error propagates. It goes to the next and to the next and to the next. And yeah, you have that what I said with the sequential composition. And yeah, the decay of your accuracy grows as n grows. So if you have a sequential multi-agent pipeline without validation, each handoff, each communication between your agents multiplies failure probability in your system. And yeah, system reliability decays according to the product rule I showed you earlier. And what actually helps is if you run a coordinated correction loop. And this is what I meant with you can do this properly with a proper retrieve loop. So what I do is I have an inner and outer loop. And one is the schema validation. So you can enforce this with Pydantic models based on the data you are using. And you get actually proper results. And let me just show you that. I hope it's not showing everything. Let me change the share and maybe I can— yeah, let me just share this window with you guys. So here you see the Palentic Docs and here just to give you a bit of— if you haven't seen this before, so you can use it from the base model and then you can do whatever data you have. You can write classes derived from their base model, and then you can basically model whatever you need. And that's the, the beautiful part here is if validation fails, then Pydentic will raise a proper error and with a breakdown that was wrong. And this is very important because then if you have, for instance, and not every model and not every model provider, and I've seen this, properly enforces that JSON and structured data, so you might even have to in between put something in place that you get the structured data. And Yeah, the, the important part really is that you can enforce it there. And let me just show you the next tab I want to show you. So, so the other loop I'm using is with instructor. And they have a retry logic with Tenacity. So, and this is just basically a Python library for adding retry logic for your applications. And that helps you to really help control API failures, rate limits, and validation errors, and so on. So you can put this together in a system. And I will go back to the slides in a bit to show you what I more mean with that. But you can have here that retry logic. And you can specify how many attempts. And very important, what you can actually do is you can also customize it. And remember, I was just mentioning with Podentic, you get feedback what went wrong specified by Podentic. So what you can do is, you can actually feed that feedback back into the model. And you can have your agent correct itself based on that feedback loop, which is here fully set up. So you get the information what went wrong, get an information how to correct it, and then your agent can actually properly correct itself with the right information. So that is the whole idea behind using those two as combined setups to give your agents actually a chance of solving the issue at hand and solving, not having endless looping and spinning and just trying to solve something from the same state where they started spinning. So this is what you see here. And I'm using those two libraries to actually achieve this. And they work beautifully and very naturally together. So what you have, one part is you have that inner loop with the schema enforcement via Pydantic and Pydantic and Instructor working perfectly together. Both is open source. You don't have to worry about that. It can handle any, any model, I'm most of the time I'm starting out with, if I'm doing prototyping and MVP and stuff and not having closed source models, open source models, I'm usually using OpenRouter, which is OpenAI compatible. And that just goes super easy into all of that. So that's a total no-brainer. It doesn't take you much time to set up and I will share also code with you how you can do this. So there's a fully worked example. And yeah, and here you see it works beautiful together. I'm very much a friend of using LangGraph because I have the full control over my graph state. I can basically set everything together with any topology, with any coordination communication as I wish. I can have my, my models set up how I will want it, also with a Pydantic, how the communication is done. So I like that to be really low level. And I put it in. And then I have my boundary gates. And then I see what— how it fails. And I have the instructor enforcement, which helps me then giving the right feedback to my agents. So you really have two distinct types of correction loops. So one is inside the model's generation window, that's for the feedback or retrieve when something goes wrong, and one is inside your system's workflow to control the data, to control the input and output of your agents, that it's properly formatted, that not, um, something which should be float is suddenly string or reverse or whatever. And Yeah, another thing what you can also do is maybe not going directly for the first generation. So not always going for a single generation, single response. So the agent can explore some multiple ones and then just go with the better results. So this is then a controlled multiple generation, not really a re-tree, but it's, I'm sure you all heard about Test Time Compute. And what it helps you with is it replaces that fragile one-shot execution with that more controlled and more robust and search-based reasoning. So you force your model to think more about those things. And you don't need always a reasoning model to do certain things. So you can actually use very simplistically, just test-time compute. And this gives you already better results. So you explore multiple candidate paths and you can apply then evidence-anchored, evidence-based ranking. And the system delays its original or its first commitment to, to that answer. And what it helps with is also resets your failure probabilities at each agent boundary. And this helps you also your agents from keeping spinning and keep looping over the task and just getting stuck and eating up the recursion limits of your multi-agent architecture. So basically it's about contracts, tools, and you get your reliable execution. Execution. So the reason prototypes fail in production is really boring reasons, sadly. You have underspecified interfaces, you have some brittle tool calls, you have weak hands-offs. Like I mentioned, this just should be a float, it's a string, or integer is a float, and then silently your agents crash. They put just gibberish and you don't really know why it And this is the case. So the goal is really you get reliability through proper contracts, through validation gates, and you get a more governed execution of that. And this is how it's ideally looking like. So this is an example of an end-to-end production pipeline. So yeah, you get that stable execution with explicit control points. So you have here user request, you have your checkpoint, you load a proper policy. So there's also a notebook I share with you, or where you can find it. And you see it here from my book. It's open source, so you can just access my repository. And what you see is there, it's, it's a governance policy, how it's structured to use your API budget. So how to access the MCP servers, how to use tools and how to go ahead in your system when a problem with validation occurs. And this happens then before tool execution and before you have any error or failure propagation in your system. So you have your policy, the agent node, you validate, okay, is the output or input valid? No, you retry, you go to the tool router if it's Okay, you can execute the tool if you have a valid result. All right, you go further, you might even want to persist the state. And then you have a final response. Otherwise, you do the error handling. And also, if you're working with, with LangGraph, always look that you from the very beginning, you have a checkpoint. And if you're moving to production, move it to a proper database that you have your checkpoint pointing in, in place. But if you have long-running agents that you actually can retrieve, also if something breaks, some API problem happens or whatever, and you have long-running agents that you can actually retrieve from the last proper state, which will save you money because you don't have to start from the beginning. So think about you have Swarm agents and they go out and about, do some research and something breaks and you didn't save the state in between, so you didn't do the checkpointing, then you have to do it all over again. So you might spend like $5 or $10 for that. And it took some time and then the system breaks. You lose the money, you lose the research, and your clients, your users will lose time because they need to start from the beginning again. So always use Checkpointer and proper storage in the database. Database. So basically, your production receive and stopping your agents from spinning is you look at contracts first, validate at every boundary in your agent system. You shouldn't use a PIN policy, and you should have it enforced with fingerprints so that you can also check this back. Then checkpointing, like I mentioned, for recovery and for your cost control. Because if you start from the beginning, it gets expensive. And also always govern tools. If you have MCPs, you still need to govern. And also some MCP tools, also some documentations and stuff. I've seen some very weird things. Sometimes, and this is actually what I did also before, I don't know if you know, FastMCP is just a Pythonic way of building this very easy. You can set up a server in like 3, 5 minutes. So very fast, it's very easy. And what I've done also sometimes to just control the token spent and all of that, and the latency is that I cleaned out with, again, with, with, um, Paidentic models on top of the, the other input from the MCP and, and run it over my defined schema and cleaned MCP and then fed it into my agents. And yeah, oftentimes this is just a better case. Sometimes the MCP servers are messy and you still want to use the service, so you build on top of that your own to streamline this. And there's not much overhead, a little bit for the first coding that, but that's about it. And yeah, I already mentioned that I very much like working with LangGraph because it gives me a lot of low-level control. And just also as advice for you, so your orchestration framework what you are choosing. This is really your control surface. And if you have multi-agent systems and you introduce coordination complexity, that's not just communication overhead, and you need to be able to have this like really properly handled. And for me, in my mind, and this is why I'm, I'm showing here on the slide with distributed state machines, so for me, multi-agent Orchestration systems are just distributed state machines. And you need to be able that they can handle the messy reality of production agents. So that means production, for production agents, your topology choice matters. So how your agents are structured, if they are centralized, decentralized, independent, or you're using hybrid, use an orchestration framework which actually gives you that granularity and you can control it. You should be able to control the state and memory. So control how your system remembers things, how you persist progress, resume execution, and things like this. And also think about your communication patterns and the rules. So how do your agents actually interact? So you can have message passing, you can have self-supervisor routing, peer exchange, things like this. So there are different communication styles and Of course, with different styles, you have different constraints how you have to govern those interactions. But none of them removes that you need to validate the, the handoffs between your agents in your multi-agent system to have it really reliable and stable. So yeah, this is just a high-level view about how agent orchestration use or works together. So this is a low-level orchestration framework, kind of overview why I advocate it. If you, if you having long-running stateful agents, really choose a low-level orchestration that will serve you better. And yeah, your orchestration layer gives you then, or should give you the choice over all of that topology. State, memory, communication, your custom validation gates, search and judges, any retrace and stop logic. So this should all be adjustable and this should lead to your desired system behavior. And this helps you then really control the reliability of your agents, your cost, your latency, and overall your output quality. And again, to go back, circle back to, to the talk's title, stop spinning your agents. So that's basically a really important choice, which you are going to have to do. And yeah, like, like you see here, so agent orchestration defines how your agents coordinate and communicate and all of that, and how they actually can recover from any errors in your systems. All right, so yeah, this is the end of those slides. This should be the last one. So just as a reminder, how you get from your demos to actually production-reliable agents. So always introduce strict validation between your agents. You should enforce the different schemas. You should enforce contracts. So really your failures are caught as early as possible and don't somehow propagate silently throughout your system. Like I mentioned with test-time compute, you can use just some simple, maybe best-of-n, just for having more paths generated, more answers generated. And then you can have a simple— simple setup, and there's a rule I will share it with you. You can actually use that very simplistically to rate, relatively rate between the different answers. And this is also important. So if you do that best of n, use it relatively, meaning LLMs as judge function generally better if they can compare the output, the answer to another one. So if they have kind of something to measure against and compare against, they are working way, way better. And then it compares and then it chooses really very reliably the best result out of that. So this also just a tip, if you're doing that, use relative judging. And yeah, of course you're, you choose your orchestration framework as you want. But I would really make an argument to use lower-level orchestration frameworks for long-running agents. All right. For anyone who wants to go deeper, so just a little bit of promotion about my books. So these are the two ones Angelina Yang was mentioning. And since I was mentioning, you can, if you want to stay connected with me, there is a Discord channel for the book. So you can ask me any questions there, or you just reach out via LinkedIn. And let me actually show you where you find the code. Let me share this with you. Uh, uh, uh, share screen. And now I can share the window because that works then properly still with that I can share more with you. All right, so you should see it. So if you go to my repository, and I see there are some comments. So I should be able to hopefully— yeah, I can't do. So I can't give you that. So yeah. Sadly, if you go there to that repository, I can't share the link with you guys, sadly. However, if you, if you put it in there, or maybe Angelina Yang, I can share it via WhatsApp with you and you can put it into the chat of people. So yeah, if you go there, if you go to chapter 5 and 6, Everything what I was talking today about, you can see there, you can use it there. And this is basically how it looks. So there's that one what is called product reliability rules. So there you can just do the computation based on what accuracy your LLM actually has in your system for one agent. And then you see how your error might propagate through your system. And here are the two loops I was describing. And like I mentioned, I'm using OpenRouter because you can literally use hundreds of models there. They recently even added Fugu. So just as a tip, if you haven't heard about it, it's a learned multi-agent orchestration system. I haven't used it yet. I read the paper, but yeah, this is kind of a multi-agent system within one model. Without you controlling it, I'm not sure how reliable it's the whole time. But since I'm sharing this with you, yeah, maybe take a look if it's interesting for you. And here then you see in that notebook 5, you see how you can build your two loops, what I was advocating for. I have some more tips in there, how you can do this, and examples. And the other, what I mentioned about the contract. So what I did was, and some of you guys might know A2A as just a communication layer between different agents or agent systems. What I did was I used that to actually use the benefits of it and repurpose it for governing the tool usage and put restraints and get reports out of it. So this is how you can repurpose the A2A framework for actually helping you and monitoring it and get it more audible for your tool usage. So this is also just a head start for you guys to govern your tool usage and govern your API more reliably. So I'm stopping if anyone has any questions. I hope it works with the communication with Angelina Yang. If she hears me. You hear me?

You see me? Yeah, I can hear you.

You hear me?

Yeah, I can hear you. I see you. Can you hear me now? Oh, I am on stage!

Yeah, yeah, you hear me?

Yeah, see me? I can— You know, I see you. Can you hear me now? Uh, I am on stage. Yeah, yeah, I can hear you.

Hear me?

Yeah, you see me? I think so. I see you. Can you hear me now? Um, I am on the stage. Sorry. Yeah, I see you. Can you hear me now? We definitely have to figure something out for this, but, um, but, but yeah, thanks for everyone for joining. If you have any questions, please just type it in here. Um, I have one last little bit of, uh, uh, information to share with you, um, before we wrap, right? And, you know, some— what I just want to share what's coming next. Nicole and I, we are running several— it's actually more than 3, at least 3 follow-up sessions in this series, and each one of them will go deeper on the themes that she introduced today. So one of them, roughly 2 weeks from now, we will talk about Inside Agents Flow. She's the creator of Agents Flow. She will walk through the design choices behind a framework that she's building. And shows what becomes possible when you wire it into your stack. After that, in August, we plan two sessions. One of the most important things that people ask is around token budgeting. She will cover token budgeting for coding agents with live walkthrough of benchmarks she's been running on Claude code. And we will also cover self-improving agent loops in a later session talking about reinforcement learning patterns and how does that stabilize multi-agent coordination without inflating your token budgets. So for details, sign up on any of the workshop page. The QR code is on the screen and Luma page will come— will be coming soon. And for the deck and the resources today, we will— as long as you sign up by any of the workshops, You will receive it by Monday. I'll shut up. I'm sorry, Nicole. I don't think I can— I can carry a conversation with you here, but I really appreciate you spending your time with us today. This is an amazing workshop. Thank you, Nicole. You spending time with us today. This is amazing workshop. Thank you, Nicole. You spending time with us today. This is amazing work. Yeah, sorry about the, the, the echoes. Um, I, yeah, thanks for everyone tuning in and, uh, we'll see you next time.