TwoSetAI Workshops #4 w. Nicole Königstein: Self-Improving Agent Loops Without Breaking Your Budget
In this 30‑minute live workshop hosted by TwoSet AI founder Angelina Yang, guest speaker Nicole Königstein (PhD candidate, author of two O'Reilly books on AI agents) presents a deep dive into self‑improving agent loops for multi‑agent systems. She explains how the advent of powerful foundation models has shifted the focus from building larger single‑task models to designing orchestrated agentic pipelines whose performance depends on topology, coordination, and learning. The talk covers the fundamentals of reinforcement learning (MDPs, rollouts/trajectories, reward design), the importance of relative evaluation using multiple LLM judges, and how to avoid reward hacking. Nicole introduces policy learning for coordination, including multi‑armed bandit‑style online policies that dynamically select tools, verification steps, and model effort based on task difficulty, cost, and reliability. She advocates using low‑level, state‑machine‑oriented frameworks such as LangGraph and her own open‑source project Agents Flow, which integrates reinforcement‑learning‑based policy training without touching model weights. Practical concerns—cost‑reliability trade‑offs, catastrophic forgetting (mitigated via KL regularization), and the need for robust harness design—are discussed. The session ends with audience Q&A covering rollouts, reward function design, catastrophic forgetting, and the future relevance of these approaches as models improve and costs fall.
Questions this video answers
Yes—rollouts refer to the full sequence of tool calls and reasoning steps the agent actually takes during execution.
Use a hybrid reward: combine a numeric budget‑aware term (e.g., token cost) with a quality score from a relative‑judging model accessed via OpenPipe/OpenRouter (the Ruler reward function). For catastrophic forgetting, apply KL‑regularization (Kubelka‑Leibler) between the old and new policy, which most modern RL libraries (GRPO, GSO) implement automatically.
Reinforcement‑learning‑based loops remain relevant because they provide deterministic, task‑specific behavior that foundation models alone cannot guarantee. Proper harness design can yield gains comparable to a new model, and RL tooling has become cheaper and faster, making it practical for high‑stakes domains (e.g., healthcare, finance) where reliability outweighs raw model size.
Chapters
Live demo. Anything can go wrong. If something can go wrong, anything can go wrong.
Of course, everything will go wrong. Yeah.
Good morning, everyone. Okay, I'm gonna share first. Hello, everyone. I'm Angelina. I think we're live. Yeah, we're live. Okay. I'm Angelina, founder and Host of TwoSet AI. Okay, thanks for tuning in today. Again, I'm gonna share the initial idea of TwoSet AI. It's TwoSet, 2 persons talking about AI, and hence, you know, I spent the last year interviewing almost 100 AI founders and domain experts like Nicole herself. I heard a lot of frontline stories and, you know, products being shipped, systems being scaled, and real problems being solved. And I wanted— today I wanna share one of the stories. One of the most inspiring stories I've heard came from Magnus Mueller. He's the CEO of BrowserUse. And before he became a founder, so he told me that he hitchhiked from Germany to Iran with just a hammock. Can you believe it? And some nights he said he would sleep in the forest in a hammock strung between 2 trees. And he told me his brain would scream with worries at night, like wild animals, where he would sleep, and whether something—
That's crazy.
It's crazy, right? Whether something bad was about to happen. And but most mornings he would open up his eyes and everything was fine. And I find that story really, really inspiring. And he said also, building, you know, a product or building a startup is about the same. Most of the fears you take to bed at night are gone by the time you wake up. So again, my goal for this channel is to help everybody here, you know, builders, engineers, anyone experimenting with AI like yourself to find your wedge, like the same way, like the, you know, the same way the founders I interview found theirs and build whatever you're passionate about and be positioned to sell it one day. Today, I'm so lucky to have Nicole again. She's fully booked. You don't know how hard to book her. These days. Um, our—
she—
and also our podcast episode, uh, was, was finally out, and I'll drop the link in the chat. Uh, today will be our last summer session with Nicole. Um, again, you know, uh, I never moved my slides. This is her picture, and she's, she's here on stage with me. I mean, I can't speak more highly of our guest. Nicole has written 2 O'Reilly books on AI agents, transformer models, and the 3rd one is coming up on harness engineering. And we will be hosting workshops on that topic here at TwoSet AI. So stay tuned. This talk will be running about 30 minutes and we'll have time for Q&A. So drop your question in the chat anytime. One small ask before I hand it over to Nicole. If you find Agents Flow what Nicole is building useful today, please give it a star on GitHub. So the link will be in pinned chat. It's the simplest way to support a project. Thanks everyone for tuning in. And please share where you're joining from, what you're building. We would love to hear from you. Over to you, Nicole.
Thank you, Angelina. And Let me just share my screen window. Here we go. All right. And let me do the slideshow. You should see my screen by now.
All right.
So today's— it's about self-improving agents loops, or in general, self-improving agents. And that happens to be my dissertation topic. So I'm, while writing books and teaching courses and stuff and developing my own open source framework, I'm also doing my PhD. So yeah, and it's about self-improving. And that's, of course, also related to the open source framework I'm building. Right. So for everyone who had tuned in earlier, to any earlier workshop, they will recognize that very same slide. You will recognize 2 or 3 others as well. However, the point I wanted to make here in general is like most of us who came from a more traditional path first with large— before the large language models and foundation models in particular were available to us, Then it was basically really that, that the model did hold much of the capability you had for your task at hand. So it was really like bigger, better, more. So you would get a better system. Now, since we have that capable foundation models and we are using agentic systems, then all of that changed. So now it's really not that you have task model output and that you have larger and more agents that it actually gets better. It can be quite the opposite because the underlying pattern changed. So kind of the equation underneath changed. So it's really the shift of a problem that you don't ask yourself anymore. And I did this very, very long before LLMs and agentic AI were here. So can the model solve that task? And now we are going to— and I'm as well, so I'm also deploying, I'm still developing myself multi-agent systems and put them into production. So I really ask myself, can my system execute that stable sequence of decisions or tasks? And yeah, the question really is not that more agents make it more stable or better. And the reason also why all of that is that if you have a multi-agent system, that kind of the architecture is defined by, by the structure topology. And also it means that one size, one architecture, one topology doesn't fit all of that. So that comes from the task type, as you see here in, in that illustration. So based on the task, you really decide what kind of system you're building. So this is very important as well. So really, the performance of your multi-agent system depends on your topology, depends on how your agents are collaborating and communicating with each other, and this affects all of what is involved. So system overhead, latency, failure propagation, all of that. Now, since the talk is about the self-improving kind of flywheel and how you then put this, or kind of put this together, so let me just shortly give you a little bit of, of a, yeah, smaller intro to that. So what we already know and did was like that from that deterministic engineering to more an exploratory search. So instead of just generating a single response, what we ended up with is that we have test-time compute, and then you're not committing to the first answer, the first result from your model. You're giving it more time to think. You have a search-based reasoning. You're exploring multiple candidates and paths. And then you're choosing the right one. And depending on what kind of task or what kind of part this is, it's the same for what I'm implementing for companies. Now, where does the learning actually happen? So what happens is you are trying something out and Or your agents are trying something out, then they see what's happening, and then they see if that outcome is good or bad, and then they can adjust the behavior. So this is where reinforcement learning actually really happens. And if you think about it, this is not so different from we learn, or how you teach your pets, or how you teach if you have kids. So So you have kind of that, all right, so see what happens, and maybe you enforce it positively, or you try to restrict and give rules and maybe punishment, but kind of you adjust the behavior based on what actually happens. Now, if you go back to the very traditional beginning of reinforcement learning. So usually this is framed as Markov decision process, so MDPs. And of what MDPs are, it's just a mathematical framework used to model decision-making in environments where the outcomes are partially random and partially under the control of that specific decision maker. And This is characterized by states, by actions, by transition probabilities, and then by rewards. So kind of if you think about it, still very similar to how we humans kind of function, right? And this is how it looks like. So this is a Markov decision process. So this is a diagram of that. So you have your states, your agents, and they interact with the environment. And I see my equations are— got spooked. Sorry for that. However, it just links to the state you have, to the actions, and then to the reward. And you update your state from the environment. And then this specific reward is used then to update your policy. And this helps then to improve your future decisions. And also what is important is that the policy is really a probabilistic decision function. So meaning you have your state and then you have your policy and then depending on if it's a task you really want to do or not, then you have the high, the medium, or low probability and or Have a preferred behavior, not a task we want to do. So really the preferred behavior. So it means that if you have a lower or the medium probability, this is actually the less favored path. And what you do with reinforcement learning, you're shifting those probabilities to the higher probability. So you go get to the higher rewards and they become increasingly likely. So it's adapting, the agents adapt its overall strategy. Now this is here just for an LLM agent learning loop. So when you collect multiple trajectories, so you have your complete rollouts then and Just if you're not familiar with that, is each rollout is one complete story of reasoning. So the whole how your agent actually reached from the initial prompt to the final output. Now, the important part is also that some might reach the goal, some might be failing, so they are not succeeding. But the important part here is every step, everything, every trajectory, every rollout you're calling teaches your agents something important about the decisions and how they lead to different outcomes. And now reinforcement learning, what it does with that trajectories, with that rollouts, with that experience of your agents is it can turn those kind of experience into structure and then you can actually use it to train your agent to behave better. And yes, since I shared the slides with you, this is just an overview for, yeah, for the ones of you not knowing much around reinforcement learning that you can actually copy that and/or use it as kind of a short-term or short reference for that and looking it up. So yeah, you have the different concepts, what it means, and why it actually matters then for your agents. So you have the reward, the trajectory rollout policy reward or judge, and then the relative evaluation and why this is important for your agents. Now you've seen that judge in one row there. So, and this is also a very important part. So The problem is that the judge or LLMs as judge do not always function that precisely or that good. So the problem is that if they are not able to do a relative comparison, meaning that you are generating different examples, different paths, And they can evaluate, cross-evaluate the different outcomes or different generations of, let's say, code, or let's say a text, then they don't really perform that well. But if you give them the variety to actually have those different paths to be evaluated, then the LLMs judge surprisingly really work very good and it's fairly solid. So if they can compare multiple candidates, then they are fairly reliable because it's easier for them to just kind of look at different examples and then rank them based on those multiple solutions. Now, this is also where kind of reinforcement learning then becomes So what you actually then do is you then shift from kind of you're not optimizing prompts and you start optimizing kind of your policies around it. And you can also optimize coordination policies. So you get your agent interactions, you have your touch model, you get your feedback, even numeric, although just also for the ones of you being new to reinforcement learning, meaning be very careful with numeric rewards and numeric feedback because your agents can be very sneaky, meaning they will hijack the reward function. So they will kind of behave if they would— wait, did learn, but what they actually learned was to trick the reward function and don't really learn that much. Sounds funny, but that's actually really a known problem in reinforcement learning. So be sure that it's a solid reward. And this is also where a church model or kind of more external feedback for your agents, for them to learn something and for the reward function where, yeah, the— Comes in very handy because that is just not hackable. So they can't hack the reward function. So this here is a high-level structure of agent acting and learning. So you have one part is the acting part and one is the learning part. So during acting, the policy produces like your trajectories and form rollout. So as I mentioned, that's the complete story of your reasoning, and then some reach your goal, some fail. And then the learning— you have that reward or church model which evaluates the rollout and then assigns a reward, and then your policy is updated from this feedback, and it's completing the learning loop. And Then reinforcement learning turns this all into structure. And you get from your raw sequences of text, you get measurable cause and effect in your application. Now, so trajectories and rollouts are really the kind of missing piece in teaching your LLM how to act. So the problem is if you think now, well, I can just do supervised fine-tuning or, yeah, some fine-tuning and then it will be fine. So that's actually not the case. So supervised learning only teaches the outputs. Then reinforcement learning, what it does, it really teaches the process. So rollouts are that entire stories of that reasoning path. So how did your model get there? What kind of tools did it call, what was the input, what was the output, all of that. If you do the supervised fine-tuning, then what happens is you have a label, you have kind of a classification. I don't say that this is not a worthwhile path for certain applications. That's totally fine. And I'm still also doing supervised fine-tuning for certain models which will become later agents in my system. But if you want to teach them to kind of learn the task they are doing on the fly and start reasoning about that and become better while doing so, then reinforcement learning is really where you should go and what you should use. Now, with that comparing, like I mentioned, comparing multiple rollouts gives your model a sense of kind of better versus worse. So this helps it become better in deciding and reasoning about the task. Now, how does this look like? So from that individual steps to a complete rollout. So you have your task, but then the policy selects next action. Your agent executes an action or a tool call. And maybe you need more steps, and then you add steps to a trajectory. You have your task complete and your output, and then your trajectory becomes a rollout. So you see here, so you capture really every step what your agent did, and then you use that to evaluate and give it to the agent to be learned. And so you have here your, your rollouts from the agent, and then for kind of getting a measurement from that, so seeing if ranking it, if it was actually good or not. So then you could use that judge model. It evaluates those trajectories, then it assigns a reward signal, and you use that relative evaluation across multiple rollouts like I mentioned. And we are selecting the higher quality trajectories, of course. So like I mentioned at the beginning, so with a probability, so we have the lower, the medium, and the high probability. Since we are wanting, of course, the higher ones, so we are of course using also the higher quality trajectories. And then we are using this to update our reinforcement learning policy. Now, this also means, and this is why I'm devoting my dissertation to that, so the kind of next phase of agentic AI is definitely more dynamic. It's compositional, and it's also more self-evolving. We've already seen things like this. I follow very closely whatever researchers are putting out there. And they— there's basically really a trend to keep them like things in the harness, also kind of moving. Let me not 100% say moving parts, but there are ways of kind of using reinforcement learning to learn the skills, to internalize the skills, to improve them. Via that the agents and make them more stable. And this kind of goes here in that direction. So think about trainable coordination. So reinforcement learning shifts that optimization from kind of an individual prompt to system-wide coordination policies. So you can learn this and you can learn where, when to hand off, verify, or stop. And I will show you a concrete example because this is what, what I do, what Angelina also mentioned with agents flow. And then workflows are kind of your training units. So complex pipelines and such as tree search or self-attach models are trained end-to-end as single structural units. So you've seen earlier on the slides I referenced 2 open-source frameworks for reinforcement learning. So you can do this with that. So you can learn those workflows as well. And then you can also have your policy integrated tool use. So tool chaining and failure recovery can be learned. So they can become learned behaviors within the policy rather than just external kind of execution calls. And now another important thing to think about when you're doing those kind of things and if you're developing your multi-agent systems, really think about what kind of framework you are choosing. I'm a huge fan of LangGraph. There's no secret to that. All my books are based Or the code is written in LangGraph. And it's not because I'm getting paid by LangGraph. I wish. No, but all jokes aside, why I'm using it is because for me it's very intuitive. Because for me, multi-agent systems are distributed state machines. And in my mind, so because agents have a certain vocabulary, so the functionality they get handed by the engineer building them. And they are having different states. They are transitioned through from the beginning, from the initial task prompt until the end. Then they finish their tasks and they have transitions in between. And this can be— well, this is long time before agents were popular, was used in GUIs and networks and all of that. And yeah, it's Finite state machines is a very intuitive way of representing agents and kind of state transitions. And also there are mechanism hierarchical state machines in modeling the multiple agents interacting. And this can be almost one-to-one modeled or used from kind of how you think about it. For LangGraph. So this is why it's, it's for me, it's fairly intuitive. However, you really want to have a low-level framework what you're using so that you can handle as much as is necessary. I'm very much in favor of being able to go low level with everything I want. I don't like, I will not use something which I click and drop together or something. I would never do that. I'm very honest about that. That's just not what I like and what I do. So I like to have my choice. And this also means you have more topology choice, how your agents are structured. Do you want, do you need a centralized, decentralized, independent, or hybrid? What are the handoffs in between? How do you verify things? How do they communicate? What's your state? What's the memory? How does your system remember? Really, how do you build the harness, all of that around your agents to make them more reliable? Then also you have an effect of your communication patterns, your rules. There are different communication patterns and based on what your application does or the end task is about, then you might favor another communication pattern over one other and also topology over another, or maybe you have a hybrid and maybe you have a hybrid communication pattern as well. But you only know this if you look into this and you match this to your task and you can actually influence that. This also means that agent orchestration This is very closely related to your low-level orchestration frameworks, especially for long-running and stateful agents, because you then can control your harness. You write it and you have kind of— I wouldn't say full control because there are still probabilistic parts. It's not all deterministic, of course, for large language models. But you can make it as deterministic as possible. Well, considering the probabilistic agents underneath and models underneath. Now, and this is the other slide I fairly often use nowadays because I really like that because it's overwhelming and I normally don't like overwhelming slides, but that one I like. So, and this is related to the framework I developed or still developing. I'm a little bit behind because I was extremely busy with putting, yeah, multi-agent system into production. And every one of you know how this is. So shortly before the system has to go live, then the people are starting testing and then there are bugs and then there are things failing and maybe the front end isn't working. And I'm not responsible front end or back end, I'm just responsible for the agents, but Then there's the back and forth between the engineers and no one wants to take charge. You can imagine this keeps you busy. However, what I want to tell you with that slide is that if you're building an agent and you're deciding about your harness, what skill, what model, what effort you're then using for the underlying model for the agent, do you verify? Do you need to retrieve? When do you stop? What kind of topology do you use and how does this affect the cost? This can be quite overwhelming and it's hard to know everything from the beginning. And this gives you kind of that challenge. So it's really hard to, from the beginning, hardwire your coordination surface because it's just like you've seen here. it's really, it's, it's just huge. It's too large, right? And even for me, if, um, with a lot of experience with that, sometimes, yeah, you, you just find things when you actually are trying things out and let this run and you need to adjust things. And this is also why I'm saying here, like, those axes influence the other axes, right? So It's not that it's isolated, and this makes it kind of difficult. And the problem is also that kind of those fixed benchmarks, if you have one, if you use one, they are not going into that surface. So there are some agent benchmarks, but it's mostly, or most of the time, it's, I think I very seldomly saw for long-running and multi-agent systems. So it's rather, yeah, okay, we got one agent and that did just a bunch of tasks, but, but that's about it. So it doesn't give you really an understanding how your agents actually perform in a multi-agent setup and different difficult tasks. Now what I did with Agents Flow, and I will later share the link with you guys, as I said, it's open source, so you You're free to use it. And I'm working on LangGraph implementation. It's already working, but, but yeah, there's still a little bit more documentation work from my side because it's fully easy and easy adaptable for everything what you're building. However, the underlying idea behind that is also reinforcement learning. So I'm applying what I just mentioned to you earlier, but now not the underlying model, is trained underlying agent, but kind of harness for everything around it, not the skill. So not the skill will be rewritten, but what kind of skill may need to be taken. So that gets decided. So I have also the chain of thought and things like this. So to help the model reason more and look into that, go step by step or just a simple one, then I have different efforts and different models. The policy here for that, so this is not much different from what I was introducing you to, what you can do with those reinforcement learning frameworks I shared with you in training your underlying model for your agent. With the policy. Now I'm using that kind of realm from the reinforcement learning to actually train a policy where how to coordinate in a multi-agent system. So to where handed off, which kind of model and model effort is better. And this is with multi-bandits. So this is a very simplistic reinforcement learning algorithm, and it just repeatedly chooses one of several actions, and it tries things out. And then based on what it's seen, it decides what to try next. And then kind of over multiple turns, it, yeah, it sees a pattern and then it goes for that. So you can also reinforce then going faster and more explicit in a certain direction, of course, or going faster. So this also means that, like, the framework, like I said, never touches any model weights. And what it also does, it learns while it's working. So I've tested this with, with Claude Code. So for coding tasks, so learning kind of the repositories and the style of your code and then choosing better model setups based on that. And it works also for multi-agent systems. So to dynamically kind of based on the task, since we always— we don't always have exactly the same task difficulty on the same task. for our application. So different users, different kind of knowledge and kind of what they give in. So maybe a model needs to reason more or be verified more and things like this depending on what kind of task needs to be done. And the policy learns that on the fly. And what it does is it has that state and it folds many tasks into few signatures. So this is how it learns. So you have folded signatures. Signatures, and then it gets regime labels. And this is the, the evidence, and it gets then the reward and gets the hands-off and all of that. It gets the belief. So it learns the intent, the difficulty, gets the evidence quality. And so it, it kind of folds this together and then that fold runs together and shares their value estimates. And this is how the policy learns. So it learns the regimes and not that individual prompts, which makes it more stable, of course. So this also means that you have your actions and actions topology becomes then your policy variable. So since we are learning on the fly during the task, And you can have this then also in production while the system is live. And then it's an online policy, which means you can learn on the fly and update it on the fly while it's actually working. It's not that you gather data and then it's offline and you train it once and have it and then you deploy it. No, it's online, so it learns while it's used. And what it does then, the policy functions sort of as a router. So instead of a supervisor and a multi-agent system, not another large language model, now the policy acts as that router, picks that action, executes, precomputes a signature, and chooses again. So this means that no trajectory out of those test cases is committed upfront. And there's also skip functions. So if something is not worth doing, It's able— the policy is able to skip that step. So for instance, if it doesn't need web search for doing a task, it just drops that. So the model doesn't even get that optionality. If we don't need a critic, we have that verifier and stuff like this, then it will skip that too. And So this means that, yeah, you have instead of a static pipeline, you have per task class and it learns then per task class. So if you have in your overall multi-agent system different qualities, so some are more intensive reasoning and multi-step parts and some are easier. And this is what I tested. Then the policy will automatically learn, and the things are more complicated, then more interaction is needed. So verifier, critic, web search, things like this, and also a better model with higher effort and more thinking and things like this. So maybe the chain of thought. additionally loaded into as an instruction for guidance for the model from the system prompt. And if it's something simple, then it kind of goes fairly straightforward. And this goes all on the fly based on what the task is about. And yeah, it's reliability aware. So it does exploitation, exploration. So exploitation is going deeper into one path which was previously explored. So we explore different ways, different paths, different possibilities, and if something is interesting, we explore it. So we are going deeper and looking deeper into this if this has merit. And this is also what you usually have in reinforcement learning. You want to achieve a proper trade-off between the exploration and exploitation. And here, kind of, this is together with reliability in the score captured. So we have what half is a failure term, and that's departure from the usually UCB. This is just a specific way of doing that, multi-armed bandits. They're what I mentioned before. And then one action, yeah, produces acceptable answer. But then if it repeatedly trips validation, then this gets pushed down so it doesn't get that high reward even, yeah, if it maybe has that acceptable answer. If it's not reliable, if it's too expensive, then it will get a negative reward so that it— future decisions don't choose that. So yeah, it's, it's really quality under operational constraints. So it goes around the reliability, and it goes around— and we'll look at, uh, at the, the cost as well. So the cost sits really inside also the objective for policy maximizers, and it's kind of that push and pull. So it looks at the quality and looks at how fast it achieves and also how much did it cost. So is it worth spending that much money or, yeah, can we have it faster, cheaper, and still a high quality? Yeah, that kind of push and pull for that. This is dynamic. And also, so I mentioned earlier, that relative judging is already better for large language models being judged because it's just easier for them to have things in comparison judged. Now, I did the same. I developed that kind of functionality in there. And what I then decided to do just to make it more stable and reliable and also Because I was really a little bit concerned about the bias and the judge from the models. So I decided to have 3 judges. And well, the reason is because maybe you have bias in one's family, and that's just not visible for me, right? So then I thought, okay, but 2 is also not good because then what happens if they disagree? Then well, I have a tie and then, yeah, spend compute on that, then the tie, I don't have a reward, right? So 3 was the choice because then I get a clear signal. So either all of them agree or I have at least 2 agreeing and then one is an identifiable outlier. So that's kind of a reasoning behind that. And if you have something fairly complicated and want to train, so just as a tip, you might want to look into this because then you're not prone to any family, any model bias for this learning. And what actually then happens, and this is fairly the same or even better for using GloVe codes here, I use that like I mentioned in the code and then run it over a week. 24/7 and on a repository benchmarking it. And yeah, basically, if it's learned and the policy decides on the fly what to do based on the task, it gets a better quality here. It spends a little bit more token, but it goes there in where you have more coordination, where it's heavier coordination, heavier, more difficult task. And then if a task is simpler, it doesn't spend that much, but the overall quality increased. And yeah, you have a little bit of a token increase. And for the Claude code, it was— I did spend lesser tokens after learn, and also the quality was better. So there was I'm really a Pareto, so I really improved the quality and lessened the tokens spent. This is the link to PyPI and paper and all of that if you want to use. For the other things, for just training your agents, your models underneath your agents with other reinforcement learning algorithms to make them kind of work better with each other. This is also an interesting path to take. And yeah, you have the links in the slides as well. If you want to reach out, that one is the Discord channel for my AI Agents book, which will be available as print in about 4 weeks. So we just did go to production this week. And yeah, LinkedIn if you, if you want to connect. And that's about it. Let me know if anyone has any questions. And let me know, Angelina, if anyone has a question.
We do have a question, but before, before we go to the questions, uh, one small ask. If you like our work here at TwoSet AI and Asians Flow, please hit like, And subscribe to the channel, give a star on GitHub. You are the reason that motivates us to keep on going, and both for Nicole and for myself. And if you have any feedback for TwoSet AI, myself, please do reach out. Our LinkedIn DM is open. Here's the LinkedIn QR code on the screen. Any founders you think I should interview, any experts you want to hear from, let me know as well. I will try my best to get them on the podcast.
Thank you.
And let's go to the questions. We have a question. When you— Nicole, if you look at the studio, it'll be easier because I starred them. The one question was by YLZM3QG. When you say rollouts in the agentic flow, are you referring to the actual runs through the tool calls and the actual steps in the agent system? As in reinforcement learning, often rollouts can be done in a simulated— yes?
Yeah, yeah, it's all the steps. Yes. Exactly.
Okay, cool. And the second question from Poljocruz2616. How do we develop appropriate reward function to update policy? Also, how do we avoid catastrophic forgetting as it is common in reinforcement learning?
So the catastrophic forgetting, so what you usually have is, and let me think, Let me just— let me share the screen here with you guys that you see it because I can't just put it in there. But if you go to that repository here, Um, there are some demos and hands-on and stuff about reinforcement learning. This is one of the courses I teach for O'Reilly, which happens to be another course, uh, next Tuesday. So if you're a member of O'Reilly's platform, um, just, uh, sign up and then you get the video afterwards if you're more interested in that. And I will explain, so the course runs 5 hours, I will explain everything and there how to get there. But in terms of catastrophic forgetting and reward function, so what I usually do is really, and you will see this in the demo as well here. And also if you go to my AI agents repository, you will see things there too. Yeah, if you go here, basically what I do is I have most of the time I have a hybrid reward. So something which is numeric, just testing, for instance, if I have budget, retrieves what the agents are allowed to do, and then I use the quality, and the quality comes from a judge. And what I use is Ruler. No, not that. Open Pipe. If it's opening, yes. So this is a framework I frequently use. This is open source, and they have Ruler, which is a reward function. And you don't need labeled data for that. It's really stable. And what it does is it actually does that relative judging. And you can assign whatever judge you want to have as long as it's an OpenAI-compatible API. So you— I usually use OpenRouter, choose a model I see fit for that task. And then you get numeric rewards from that and it evaluates the quality of that. So like I said, I combine like the maximum retrieves, maybe I do punish the amount of tokens they are using. So I did train, for instance, a model to do the search with EXA better. So I, punished, then too long searches, then it was spiraling, things like this. So I combined this. And for the catastrophic forgetting, so what you want to do is that, or what actually happens if you're using reinforcement learning, and this is one part against that catastrophic forgetting, is you're using Kulbak Leibler, and this is regularization. And this means basically just if you think about your old model and— or your model at a certain state snapshot, and then you're having that policy and you're updating the model's weights with that policy. And what Kulbak Leibler actually does, it evaluates the difference between the old policy and the new one and kind of pulls it back that it doesn't take too heavy steps, too big steps in improving the policy. And if you do that too fast, if you don't use that, that Kovacs-Leibler, then you really would run into that definitely into that catastrophic forgetting. But the Kubelka-Gleibler actually helps you. And the standard algorithms with GRPO and GSPO already use that. So usually you don't have to worry that much if you're training kind of that and using, yeah, the correct frameworks implementing this. I hope I hope that answers her question. Rather lengthy answer, but yeah.
Yeah, there's one more question. How much longer do you expect this approach stay relevant as the models are improving over time and the costs are going down? By Chen J, 2007.
In general reinforcement learning, I would say we will never have At least I don't see it in the foreseeable future that a model will always, or will be capable of every task and be, be very good at everything, right? So the thing is, when I do reinforcement learning, when I'm training the models, is with reinforcement learning is really like If I want a more, and I quote here, deterministic behavior, because I am training the models how to, for instance, behave within a multi-agent system in a specific role for a specific task. So I'm diverging from it being that foundation model, that generalist, in having it more specialized to what I want to achieve. So being more kind of saying now more considerate, but better in the teamwork with the other agents, be better at that specific role, at specific tasks. And since we have that foundation models, of course, they have a large amount of knowledge about the tasks, but you want them to be specifically better at that specific task for you. And most of the time, you, you will achieve a bit better result. Although of course I would also always, before I start training my model, I would always improve my harness. And there are research out there that if you have a proper harness, if you, if you have it properly set up, uh, that this can amount for some cases as much as the next model update and the model getting better. So you need to weigh the— if it's— is it worth or not? But also the thing is that the training with reinforcement learning, with all the frameworks out there, is getting cheaper and easier. So like 3 years ago, it was really hard doing reinforcement learning. Now you have frameworks with OpenPipe and RLLM where it's, it's really easy to do, and it's not that costly anymore because it's optimized with those frameworks. And you can, let's say, in 1 or 2 days, depending on the task, even faster, train your model with reinforcement learning to be better at this specific task. So.
Yeah, this sounds to me a very, you know, philosophical question, Nicole.
You know, the real world— Sort of, right? It depends on the quality you need, right? If you, if you're in healthcare or finance, it might be worthwhile. If you are in a domain where a small error might be okay, although even, even for the companies I'm I'm implementing multi-agent system, they are still very, um, how can I put this, um, very strict on errors, even though it's not life and death or cost like a million or billion or whatever amount of money if a model is wrong. So they're— I still want them to perform, but because it goes out to customers and customer pays, and if customers are not happy then it goes down that chain, right? So they lose customers. So you might want to invest some time to train the models after all.
Yeah, thank you, Nicole, for today. And thanks everyone for joining. And before we wrap up, just to let everybody know what to expect, this is our last session for the summer, but we are planning for more sessions in the fall, and please sign up for our newsletter. We will communicate through the newsletter, and Loomer calendar will come out soon as well, and I'll let everybody know. Thanks for joining again, and thank you, Nicole, as well. Again, I'll end.
Thank you.
More from Angelina
The MrBeast Rule That Quietly Decides Who Wins on YouTube | Rox Codes, CEO @ Flightcast
Ask Any C-Suite This One Question About Their AI Agents | Hector Hernandez, COO @ Speakeasy
