Inside AgensFlow: The Reliability Layer for Your AI Agents — Workshop #2

Me hearing you now.

Yeah, fantastic. We're finally live. Yeah, do you mind if I just double check? No, no, no, go ahead. Yeah, apologize for everyone. Thanks for tuning in. Give us one second. I think we're live. Awesome. We're good. Perfect. Yeah, we're good to go.

And you could see my, my screen also, right? So my slides were visible. Do you see that? Can you check?

Oh, can I see what?

The slides.

Moving. Oh yeah, yeah, I can, I can see you. I can see your slides. So I will, uh, so I noticed that it will, this will actually, you know, hide your slides sometimes, but I'm paying attention to the stage, so I will, I will take care of it. Um, so, so you can keep talking. So, okay, okay, let me kick it off. Um, hello everyone. I want to say good morning, but I think it's a good evening for Nicole and, uh, probably our audience as well. So wherever you are, hello everyone. I'm Angelina Yang. I'm founder and host of TwoSet AI podcast. Thanks for tuning in today. I have a little bit of spill to share with you today before we introduce you Nicole. I want to share a little bit about TwoSet AI. The initial idea of TwoSet is two set, two person talking about AI, right? Me and Nicole, we are two person and we share a lot of interest and I'm so happy to have her and chat with her. But I started this Thinking, instead of showcasing single prototype ideas by myself, what if I just ask the experts, the proven founders, the builders, just like Nicole, to share what they're seeing actually in production? I spent last year interviewing more than 100 AI founders and domain experts, and I have received thousands of questions from the audience. I've decided that I'm going to set up this series of workshop that I can get these experts, builders directly in front of our audience and sharing insights of real production pain points and how they solve it and answer questions on the spot. I also recently, for audience who have been tuning in, really appreciate your support. I also recently sharpened the focus of TwoSet.ai channel. My goal is now to help builders, engineers, and anyone experimenting with AI like yourself to find your wedge the same way the founders I interview found theirs and build what you are passionate about and be positioned, be ready to sell it one day. Nicole was on the show and our episode will be released. I'll share with everyone today. Today Nicole will walk us through Agent Flow. The open source framework she's building to fix one of the biggest pain points in building AI agents, the multi-agent system that works in testing and breaking production. Last week she gave us the math of why they fell and all that. Go back to tune, have a replay if you missed that one. Today she will show us the framework. I cannot speak more highly of our guest today. She's written two books with O'Reilly. One is The Definitive Guide for Building AI Agents. And she's writing a third one on hardness, so stay tuned on that one as well. I'll definitely invite her back to talk about that one. She's also teaching at Oxford, O'Reilly, and other venues, and now my honor to have her on Two Sided AI as well. So the talk today will be around 30 minutes, and we will have time for Q&A, so leave your question in the chat anytime. One small ask before I hand it over to Nicole. If you find Agents Flow interesting today, please give it a star on GitHub. It's the simplest way as a fellow builder to support a project. Well, thank you so much for joining today. Yeah, before we get started, please share in the chat if you can, what you're building on, where are you located, and we will love to hear from you. Let's get started. I'll hand it over to you, Nicole.

Thanks. And thanks for the kind introduction. So let me get started. So it won't be all about Inside Agents Flow in terms of the framework itself. It's kind of also why I even started out doing that and a little bit of a reasoning behind it so that, yeah, you get all better understanding if you're starting out building agents, what problems you're most likely going to face if you are scaling this and how to think and how to reason about those things. And yeah, and the reason basically why I then started out building that, I'm still not done, I'm still building a lot. So it, there will be a lot of more releases. So that's just the first wave. I just committed earlier, like, I think like 20 minutes ago, another release. So yeah, in any case, stay tuned. There will be many more. All right, so Angelina already told me, introduced me, so I'm going to skip that. And so this is kind of a repeat from the last part. A few slides will be repetition from the last talk or workshop we had. So just that In any case, if someone wasn't attending that, that we are all on the same page. So generally, the problem is that there's kind of a perception that, well, if you have a capable model that like large big models and they're, yeah, that your system will be just working perfectly fine. And then if you tie, of course, more agent, more models in terms of then more agents together, then it it will be even more awesome and it will just work perfectly fine. But that's kind of the wrong impression. So it really goes from that kind of decision point. So can your model actually solve that task at hand to can your system execute that kind of chain you set up as a stable sequence of a decision in your system? And this is really what way over almost 3 years started out bugging me when I started building agents. So yeah, and another important point to think and reason about agents and multi-agent system is really the difference between communication and also coordination. So communication is the message exchange between your agents And then the coordination is your strategic control of activity. So that is really important to, to understand. And then also a very, very important point is we have different topologies, right? So we have different architectures because one size doesn't fit all of your tasks. So depending on your task, and I'm, I'm specifically saying task, I don't mean use case. I mean, really, you may have a use case of an application you have in general in mind. And of course, your, your topology, you might have topology which works, and you decide it. And this is also how I teach it and say, right, you need to think about what you're trying to achieve, and so on. But maybe the point is that in that Some part might have still to change depending on the task within the overall use case you have. So keep that in mind. I will get back to that later during the talk. And this is just to give you an overview for what kind of different topologies, classes we usually have. So we have that centralized part where you have that one orchestrator. You have different subagents. Sometimes it's— the term used lead agent, or you have that supervisor. So that goes in that direction. So you have your centralized multi-agent system, then you have it decentralized, like with SwarmSphere and so on, and the same with independent MAS. So they are more on your— on their own and exchange the data differently. And then oftentimes you have really the hybrid one, which means you, you have one or two of those, or let's say we have three of those here combined, or just two together. So how you have a hybrid system. And as you see here on the slide, of course, the, the point always is that the topology distributes the communication and the coordination then differently. And there are also communication patterns where you can also adjust things. I won't go into the different communication patterns here, but there are still things you can kind of then look into that. All right, so now a lot of the talks, and Angelina just mentioned it, I'm already started, I'm writing another book for O'Reilly, so my third book, and this will be about harness engineering. And so this is also a topic I'm highly invested in, and this is also related to agents flow and in general, the research I'm doing, and of course, some implementation I'm doing as well. So, and this really like kind of, yeah, underlying question. So how do you scale the harness and how do you learn actually what matters for your system? So, and I don't expect you to be able to read all of that. You'll get the slides if you're— you can just connect with me on LinkedIn. I sent you the slides, the PDF. No, no, no problem at all. And yeah, basically this is just to give you an overview over the Hardens and how it widens the scope. So we all started out with that prompt engineering. So you optimize for input, you have that single large language model, maybe a little bit of tools. And then you optimize that. Then the next part was, all right, so then we noticed, well, we need to optimize what it sees, so compress for context, get RAG into that, and really optimize what the model actually sees in its context. And now the next step where the research and also the industry arrived is that harness engineering part. So you're looking at the whole system and you optimize actually how it runs. So you keep— and for all of those steps here, what you see is— and this is why it's here, the red rectangle on the bottom— is the model stays fixed. So you have fixed weights. It's unchanged across all those three columns. So that's also important. Keep that in mind too. And this is about the general— the— anatomy of a harness. So you have different components, what you're looking into, like we have the identity, you have behavior, you have the runtime, where you have your different tools, some adoption verification and observability and governance. So this is just to give you a better understanding if that kind of topic is new to you, that you get a bit of an understanding what, what all goes into a harness and shadow org. All right. So given that, now that we kind of established that prerequisites and understanding the harness, so what actually happens is that your harness turns execution into control, right? So you can plan, you can execute, you observe, you verify, then you get correct, and then you can do adjustments. And all of that does not involve to change anything in the model. So you're not training later the model. You're not altering the model at all. It's just a harness. Now because of that, the control problem moves to the next decision, right? So then you have the execution differently, and you're not altering the model. Now you have the next step kind of selection becomes more of that harness control problem. So how do you do that? What are the components? What should get altered and exchanged and so on, or learned? Now, I really like that slide and I use it a lot because it's, it's just really confusing. And this is how I feel or felt sometimes when I started, uh, when I, I tried to find the right combination for multi-agent system. So So you have so many choices. Like when I started out, like I started out in 2019 with Transformers. And for anyone who also started that early, you might remember we didn't have that many options in large language models, right? So it was really narrow. Now you have so many options and It's not only that you have a lot of different models at hand, you have skills, you have, you can adjust the efforts, how much thinking should this model actually do for the task? And then we have that verification pattern and we have RAG, so retrieve, and when should we then stop all of that? So for the effort, maybe we have thinking tokens. So when do we stop thinking? What makes sense? What kind of topology do we choose for our multi-agent system? And all of that affects, of course, the cost. And of course, somewhere in all of those different setups, there is for sure the right solution. But yeah, good luck finding that and just engineering it all up front.

Nicole, can I ask a question? I think there's a really good question from the chat. So people are asking if you can define some of the terms. Is, you know, the top, at high level, at least like topology, harness, some of the contextual technical terms.

So topology is just how you set up your, your agents. So if we go back here, so this just basically means, so if you look at that centralized mass, that first figure. And this basically means how your control flow works in your multi-agent system. So think about— that analogy will, will help to explain that. So think about you're working in a team, and you're a developer, you're an engineer. And then you have your tech lead, and the tech lead gets an input from the manager. And then that tech lead decides to whom of one of the team— here we have just 3 engineers, so we have 3 subagents— so to which one will that tech lead give a task first. And this is what, what the topology means. So this is just the architecture, how your, your multi-agent system, how your agents coordinate. And you can think of your agents just kind of as workers, and you give them work packages. And then depending on what kind of architecture or topology you're using, the work gets done in a different sequence or independent, and then it gets aggregated. So things like this, I hope that makes it clear. And the harness is, is the— let me go here around— the harness is the infrastructure the layer you built around the model to make the model do the things more reliably. So you have your tools and the environment. So basically, an agent is the model plus the harness of the whole infrastructure you, you built around. So that's not the topology in that sense, but the tools, the environment you give it. Then, of course, part of the orchestration. So do you give it a planning? Do you give it verification, retrace? So if there is something not properly done from one agent, how do you handle the loops? So do you have 10 loops or do you have 4 loops for that? You have then the verification. Do things go correct, do you have governance and security, and you eventually will look at how your harness performs. And then you will adjust this based on your results. And you will not adjust the model, you will adjust the harness. And there are actually results already that if you are adjusting, modifying the harness properly for your use case, for your tasks, then you can get better improvements of the overall accuracy of your system than just getting— or more improvement than from a next model iteration. So usually the next model iteration from a flagship model provider does improve the overall success of your overall system lesser than if you're optimizing your harness. So you can use, let's, I don't want to say cheaper, but a simpler model or can stick longer with not the latest, latest, latest model, but still have reasonable, good, good results of your overall system or agent system. All right, let me go back to that. So yeah, and that's kind of a part also what I mentioned. So the agent is the model plus the harness. So And that's really the point and what I kind of tried to make while I was answering the question. So, reason is really that if you have a wrong setup of all of that, a really good model, a really, like, expensive, really good model can still go off the rails. So, you have bad decisions, bad planning, wrong skills, weak retrieval, things like this. So then it goes off the rails. It tries to fulfill the task at hand, but it just can't because it doesn't have the right setup. It will just burn your credit cards. You will— it will burn tokens and it might not perform the task as you wished. And it might, might even not perform at all. And this is really why bad harness engineering. Yeah. You might get the wrong answer and then you have just those runaway costs. So this is why the whole engineering around your model is super important. Right. And this is also a recap from, from the last workshop. So just a reminder that if we are chaining both systems together, that we, we have the dependency of how good the previous Well, we have, even if we have a very good previous result, then if we chain multiple things together and then maybe there are some inconsistencies in between, then we are still relying on the total system success on how good then the overall chaining works together. And if we, if we don't control that error space, if we have error or failure propagation, replication, then we have an overall loss of the accuracy. And yeah, that's just a fact. And it, that law is, is, yeah, from the 1960s. So it's pretty old and it still applies to AI agent reliability. And this is always a recap. So just keep in mind that you can mimic that system or agents are reliable. You can have a lot of retries, but the retries not only cost you tokens, it costs you time, it costs you latency. I have here 11 attempts to get from a very bad success rate to a high success rate. This also then gives you the problems that you get a high tail latency, meaning that a certain part of your users will have a longer time to, to wait to get the final answer, which then feels to some of your users that the system is broken. And then yeah, it's of course bad, bad for your, your application. Right. So then another point I want to make is that in general, the composition of your system is not neutral, right? So strong agents do not automatically produce reliable systems. So this is what I mentioned So that's a model. So underneath that agent is a strong model. So you have a strong agent, but this does not automatically magically mean, well, everything will be super. And this is the point here. So each agent can be locally capable, but then the chain can still be globally in Korean. So that's part of the point I made with the Thalassa's Law. And then because of that, the reliability problem lives between the agents as well as inside them. So it's not just the, the probability and the next token and how good your model is at a certain task with a benchmark. It's really the overall system. So, and just some math, and that's the last one. I, I won't bring more up, but this makes my point fairly precise. So your overall system success is not dependent on the success of your individual agent. It's more like how many steps and how many agents and then how good they communicate. This is just a small part. There are actually more parts included in that, but that would have been a very complicated formula. But just to give you an idea, so how good they are communicating, so how good their, the handoffs are between that, that this determines also how good your system will be. So if they introduce errors, then the next agent will build upon those errors. So this is why an agent handoff is not a function call. So the receivers of the next agent, this is still a probabilistic model, will interpret the message And it does not merely execute it, right? So you see here on the one side, the tool boundary, you have your structured input, some deterministic semantics, and you can even test the results, usually, if it's not some, some web search, but then you can still have a citation, you could do mathematically, you could look at the, What's the word? Euclidean embeddings. You just can measure how close the embeddings are from what you found and then to the query and all of that. So you get still a sense of even if it's free text from internet research, you can still have a way of determining how good your result is. But with agents, then maybe have partial task representation, then you have a probabilistic interpretation, and then you have some local trajectory. So this is why it's really totally different. And yeah, so a sub-agent call is not a tool call. So really it introduces another inference process, and this comes with different challenges. So tool calls and agent delegation are different boundary types. And your schema can validate the structure, but it can't guarantee that it preserves the meaning overall in, in your multi-agent system. And now this is one of the parts what really bothered me, and which is why I dedicated parts of my research to that. And so what mostly happens is that benchmarks and all those model providers, well, they show you, yeah, but it succeeded and it does this and that, or most of the benchmark scores. But you have that model capability measured, and that is kind of a single out measure. A lot of those for harnesses, like single agent harness, and it's— it doesn't get measured how they are coordinating with each other. Is there communication loss? Is there an information loss? Is there anything what might be interesting to understand how you should route it and things like this? So there's no identification how good your multi-agent coordination actually works towards your final outcome. And if you could get there probably cheaper with, with another setup. So that was kind of my trigger. So I did do some research there and I tested some things. And yeah, this came down to kind of a conclusion that coordination really needs its own signals. And you can't improve your agent coordination. So your topology, your agent architecture, what I explained before without having proper feedback. So with output-only feedback, you're not able to do that, right? So you need a bit of a better understanding, handling of communication in between, and then map this also to the final output. So that kind of goes then from agents in general to arriving at more trainable agentic program. So once you are composing your agents and you have your coordination trajectory, then this becomes a surface what you can optimize. So that was my thought behind all of that. So you get trainable coordination, you can learn how, how, um, where to end off, verify, retrieve, or stop, because it doesn't make sense to send it to that agent at all. And yeah, you have workflows as training units, right? So you can evaluate those complete execution structures. You can have policy-integrated tool use. So you can look at both routing, chaining, and so on. And also what you get on top of that, you get trainable harness artifacts. So your skills remain external and fixed. And still you get an expectable, you can optimize, you can even— and this is actually what I did. I also looked at using different skills with models and how then the models performed for different tasks. And this is something you can optimize as well. So now, stepping a little bit back to what was already introduced— let me think— over 2 years ago. So we had that, that we don't need for everything that extremely large model. So we could also just use a smaller model and give that smaller model more time to think and do actually the task. That's the same with humans again. So if you, let's say you are, on Jeopardy, and well, then you might be nervous. That comes on top, but you have to answer in a very short amount of time, right? So more or less instantly. So then if you are at home and you are totally relaxed and you have probably when you just mute or don't look at what the answer were, you get more time to think about it. You can reason about it and then think about different paths and then come up with the final answer. And most likely you, you will find the answer as compared in just a talk. And that's the same with the models. So if you give them more time to think about it, they usually can outperform the larger models even while being smaller models. So this is kind of a revisit what we did before. So we, we froze the model, we AFA test time compute, and then we got it better. So now we kind of do the same. So the model remains frozen and the trainable state can move from the model weights into your harness. So you have your frozen target model, you can adjust your harness, get your execution, you score your trajectories, and then you learn how that worked for your system. And now if you look at the harness and think back, well, what I showed you with the overview, so a skill obviously is one part of the harness, right? So that is one part what you can think about giving it a try to learn or route and decide which skill to pick. Then roles and capabilities are also harness decision. It's not that the model decides this is, this your harness, this is your system, your infrastructure, infrastructure you build around your, your model for the whole system to work for your application. So you can put the decision of which role should act next, using what kind of model, what model effort, and what skill to attach to. You could— you can outsource that to the Harness as Harness decision. So as you see here, that's just examples from parts what I tested. So now what I said before with the use case and the topology or your overall architecture of your multi-agent system. So that's the same. So the topology is also a harness decision. So topology becomes or can become an action, not a permanent system identity. So meaning what we usually do is you set up, you engineer, you think about, you reason about the setup of your multi-agent system based on the use case you have in general. And then you choose that what the most reasonable architecture setup is for that. So the multi-agent system best for that. But I already mentioned, so maybe task in between differ a little bit. And maybe for some cases, you don't need all the effort, you don't need all, all the agents, you don't need the verification, maybe don't need memory and things like this. And of course, now some of you might say, Well, but didn't you just told me, like, with that centralized multi-agent system, I have my tech lead, my supervisor, my orchestrator, and that one decides where to route it, to which agent, to which engineer giving that task. And there are different ways to look at that. So one point is it can only route then what agent, and then you still don't have the surface of the skill attached and the model effort, so that supervisor doesn't select that. And also another point is, well, you can't actually see or observe and audit what the underlying decision to that was, right? So that, that's a problem, and it might not be stable over all tasks because it's not learned. So it's just, it gets the task and it routes and sometimes it routes there and sometimes it routes differently. Now, this is why it's super hard to fix the multi-agent coordination upfront because that still differs, right? So I might, because I'm having experience, I might be really good in having a good system set up and it it looks fairly good. But yeah, like I said, for some tasks it might not be the optimal choice. And still looking here on that slide, so we still have a problem. We have skills, model, effort. Do we want to verify it? Do we need to retrieve? What kind of topology do we really use? When do we stop? And this actually then influences the cost, and it's really hard to get there, like that, that golden path to that funny looking robot. So yeah, that, that was basically the, the question, what was plaguing me. So then I thought, all right, so why don't we go and learn that? So really going from fixed design to learned coordination. So instead of having an engineer decide upfront, hardcode the choices. And still, we have so many choices to make. And then there comes a new model. And now do I actually wire that new model into my system? And oh damn, I didn't have any evaluation data. I didn't do that. I don't know how to build the evaluation set to actually know if I should use that new model, but let's say your, your manager says, well, but there's a new model, why don't you deploy that? And all of that. So there are a lot of problems arising from, from all of those choices. So what I then did is I created AgentFlow based on that problem and questions I had. So it learns not only the coordination policy, but you can basically learn through repeated runs, um, what skill, what effort, how it should coordinate, when it should verify, and all of that. So you learn the, the coordination part and, or the, the surface of that different parts of coordination skill and so on. And this also is then adaptive and it's auditable and it's also transferable. That is thing I found very interesting was if you have similar tasks, and I actually tried that, so I transferred it. So actually, that is possible. So if it's relating, you can have it trained once. So you— there's still an upfront cost, you need to train it, you need to learn the coordination with the policy. But And then also a very good benefit is that— and I tested this too— so if a new model comes out, you can just wire the model into your system and have it deployed and then let the policy on the fly decide if that actually is a better choice. And if it's a better choice, it will route it more frequently to that. Model. If it doesn't pay out, it will just ignore eventually the model, and you will not use the new model. So this is just an example of a task I, I did. And there are different classes. So you see, these are different coordination-heavy and retrieval and verifier-heavy tasks. For a security task and distributed system task. So I tested those two things, those two tasks. And yeah, you see for C1 and C5, which were not so coordination heavy, there it didn't perform better. And for the other ones where it was more coordination heavy, it actually learned the policy and learned really the right choices. And it doesn't skip forward. And this is what I meant previously with the very interesting part. What I found was that actually that this becomes transferable. So I tested this. I first had the distributed systems task and then the security incident task. And then I just transferred it. And so you see here the warm start, and it shows you that the warm start helped in kind of jumpstart the quality and also reduce the overall token cost for that. So the learned policy was transferable to another task. And another interesting experiment I did, and this worked also, very interesting results actually, was I used it for Claude code and also different coding tasks. And there were different, 3 different charges. So for the previous ones, there were also 3 different charges. And I had like token and how often it needed to retrain and things like this. So in the paper and in the repositories are all the, details, but basically, um, importantly, I had 3 different charges. And the reason why I have 3 different charges is in the system, so there's no single biased charge. I have also the charges from different providers, and of course, because of different providers, there are different models. And so not a single provider, so not a single provider bias, and just using 3 different models. No, I really designed it to, to be more independent and 3 judges. The reason there is that if you have 1, you have that single judge bias. If you have 2, well, what are you doing when you have a tie? So 3 is the reason. So if, if I have at least 2 out of 3 agreeing on that the task was properly done, that the communication, then the things between work properly, then I have a strong signal and I can use this to actually train the policy and I can use this to actually judge if the result was good or not and then learn from there. And what you see here is actually it find— the policy found out by itself that some the coordination, it needs a stronger model with more model effort. And for the simple workload, you can just use the cheap model and you can have a Sonnet and in between. So it really distinguished on its own while doing those tasks and repeatedly doing those, seeing and learning that, well, for the planning, then we will have that stronger model. And then for build and verify and review and ship it, yeah, we don't need to spend that much money. That also suffices. And also for the ones of you experimented, and I'm sure every engineer by now experimented with either Codex or Windsurf or Cursor or Load Code. So if you are deciding to stay with a model with a high effort. And you, you know, you see that it starts like trying to refactor your super perfectly fine code, overcomplicates things sometimes, screws up, makes errors, and just because it overthinks and overdoes things. So properly, yeah, correctly learning where to put more effort in actually improves the quality, and this is also what I've seen. So the quality improved and it was agreed upon all those 3 judges and it also cost lesser. So this was also a very nice thing to see. And also for the other task, and you will see in the repository, I have notebooks and for that I just run earlier the task. So what actually happened was that the performance of the learned policy was better than a very good model and always high effort. And it was like almost 50%, so 45% cheaper if you would go with the always high model in that multi-agent system. This was not for coding. This was just also like a security incident task. For multi-agent system. And yeah, it was faster, so less latency, and it was cheaper, and it was better. So these are kind of, these are kind of my, my current results and the reason really why, why I did come with that. So if you want to use that, and yeah, just with the note that I'm still developing, You can find all of that there. It's also published via PyPI, so you can just install it, PyPI. And yeah, that kind of concludes, and I'm pretty good on time, I think. So yeah, basically that's from your interface, your harness to getting to adaptive systems. So you get your executable reasoning with your models, you get your programmatic actions, you inspect your states. You have your feedback-driven control, and then you can learn your harness decisions from there. And yeah, if anyone wants to connect and get the slides, let me know. You can connect with me. And if anyone has questions, I'm sure Angelina Yang will let me know if anyone has questions.

Yeah, type in the chat. Thanks, Nicole. This is amazing. Alright, let me, let me, let me switch this to a little bit wrap up. So thanks for, thanks for everybody who tuned in today. And before we wrap, some quick words on what's coming next. Our next session in the series, you know, will go deeper in some of these different aspects that Nicole covered today. The next one will be really, really, really interesting. It's essentially from token maxing to token budgeting. So if you are thinking of making, you know, making things work for you, not just reliably but also economically, tune in for the next one as well. This is also something that Nicole is working on to enhance her agent's flow. And she will speak about how to make that happen. Then after that, we also have self-improving agent loops without breaking your budget, budget. That's the reinforcement learning patterns that stabilize multi-agent coordination without inflating your token bill. So full details are on our workshop page. The next session is open for RSVP now. I'll share the link The QR code is on screen. This one is for the whole workshop series. The next one, the fourth one, will come up soon. And again, also connect with me on LinkedIn if you have any feedback for our session or more guests that I should invite. Let me know and we'll go from there. Yeah, hoping you enjoyed it. Thank you so much, Nicole. Always great to see you. Yeah, okay, awesome. I think I'm gonna close out this, and I'll end the stream now. Bye everyone, see you next time. All right.