Token Budgets for Coding Agents | Nicole Königstein

That's okay.

That's totally fine.

Yeah, let me see if we're live yet. Great, I think we're live. Hello, Nicole, great to have you here again.

Hello, Angelina. Likewise, happy to be here.

Yes, thank you. So welcome everyone. I'm Angelina, founder and host of TwoSet.ai. Thanks for tuning in again if you've been here twice. I hope you enjoy our last 2 sessions with Nicole, our speaker here. Before we dive in, I want to just share my little backstory about TwoSet.ai a little bit. I started about 2 years ago and I shared a lot of the tutorials before I started interviewing founders and domain experts. And I did the initial idea of Two Cents AI is 2 persons chatting about AI. I think it's more fun to share ideas and chatting. That's why I have Nicole here and many other founders. And I spent last year actually interviewing more than Well, almost actually 100 AI founders and domain experts, including Nicole here, and received a lot of questions from the audience. I decided that why not setting up a series of workshops so I can get actual experts here, the builders who are building in the frontline, solving real production pain points and share their ideas, insights, and you can ask questions on the spot. I also recently sharpened the focus of my channel. My goal is to help builders, engineers, and anyone, because everyone can vibe code these days, right? Anyone experimenting with AI like yourself or your colleagues, your friends, to find your wedge the same way many of the founders I interview found theirs and build something you're passionate about and be positioned to sell it one day. Nicole, again, was on the show and her episode is almost ready. Nicole, sorry for the delay. It's going to come out in the next 2 weeks. Today is our workshop. Today is our workshop 3. I'm on a series with Nicole. I'm not gonna spoil it right now, but I can't speak more highly of our guest. Nicole has written 2 books with O'Reilly, and she's writing the 3rd one. And I have the book covers here so you can see. It's Definitive Guide for AI Agents. and Transformers. Her third one is on harness engineering, so it'll be really interesting to hear more about that one. This talk will be about 30 minutes and we have time for Q&A. Please drop your question in the chat anytime. One small ask before I hand it over to Nicole: if you find what she's building called Agents Flow useful, Today, please give a star on GitHub. The link will be— the link is in the QR code on screen and I'll pin it in the chat. It's the simplest way to support a project. Thank you. Thank you so much and thanks for joining. And please do share in the chat, where are you located? What are you building? We would love to learn more about you. Yeah, over to you.

Nicole, thank you. Thanks so much. Yeah, and let me start the slideshow. You should see the screen now. All right, so yeah, so the agent's flow is kind of different parts what you can use. So one part is you can use it for your own multi-agent systems where you can save tokens and make it more reliable. That was part of the last workshop and what I then did and tried out. And you will see things today and you can— it's open source, you can do it yourself. You just have to connect MCP and you will see this during the talk. So basically, in general, the question on this was a similar slide I have last talk. I think it was the last one. So what, what still bothers me and strikes me, and there's another slide I have in there, is there's a little bit of a misconception that bigger is better and more, so more capability and all of that, more model effort makes all of your system or the result better. And if some of you come from a longer history of, yeah, AI, like I do, when I started out, of course, transformers and all of that wasn't there. And I had to build the models according to the task. And the problem was that, or kind of that, all of the capability of the task, what you were trying to achieve, was mostly in the model. So the model needed to be the best. You basically built a lot of capability into the model. Funny enough, I recently just saw an article from one of the fellow authors from O'Reilly for their Radar. He was mentioning something about running voice agents on a Raspberry Pi. That reminded me, and this is why I'm telling you this, I was like about 10 years or so, I had a Raspberry Pi or had several ones and I installed a model and had just a few lines of Python code to get the sensors running and stuff like this. But I literally had a robot which was able to move around my floor, around my apartment. It has a camera, had a camera attached and sensors, LiDAR and stuff so that it could navigate. And yeah, the surrounding code, so the, if you want to call it the harness around the model, that was really not that much. So what I did, was all the capability and training was really in the model itself. So now this is not the case anymore. So you have— we have like that switch that we have those foundation models which are already highly capable. So we usually don't have to train them anymore or fine-tune them or adjust them, whatever method like supervised fine-tuning or reinforcement learning. So now you, you need to engineer the system more around it. So this is why I'm seeing that. So that was the previous assumption, and kind of some of that is still tracked through the time now where it really isn't really the case anymore. So that's just the point I wanted to make. And given that, If you are working with, yeah, Claude Code, Codex, Cursor, I'm sure most of you or all of you are already aware of that, that coding agents are just loops, right? So you have propose, act, inspect, correct, repeat, and so on. So you have your loop. And then there is just that thing that you with Claude Code, You own the coordination layer, so you don't own the inner loop. So Janila mentioned that I'm writing a book about harness engineering, and that will be really about the harness engineering for multi-agent systems. So where you as engineer really are engineering and owning the whole system. So you set up your agents and your own— all the verification, the loops, the graph, all of that, you own it and you offer it and you engineer it. But when you are using those coding agents like Collode Code, you don't own that. So you don't own that kind of inner loop, how I call it here. But what you can do is you can use MCP. So they are— it's easy to wire that. You can have hooks and skills and task decomposition and so on. And this is also a harness, and this was the thinking behind it. So I first kind of experimented and also developed Agents Flow for normal multi-agent coordination because of course there it already bothered me that we have to set this up from the beginning. And, and it kind of was static. And even if the task would vary, that it would just route the same, even if you have supervisor and an LLM doing that, it's, it's still fragile. And it, it's sort of non-deterministic because I had a lot of problems with that. So that was just a reason why I invented this or created this, and I wanted to solve that problem, which drove me mad sometimes. Now then I had the idea, so, all right, since it's working there, maybe it could work for coding agents as well if I can get enough access to those points to actually alter things and get like access to the loop. And as it turned out, that is actually possible. So for that, so we are doing loop engineering and you can do loop engineering separately if you are not owning your harness, your agent harness. What's the case with coding agents, right? So you're not using— you're not owning that. So basically what is the case here then? So you have the user task and do this and that. Then you have your coordination harness. So you have the model routing. The skill choice, the effort, the budget verification, if it needs to retry, do something again, and so on. And this can be learned, or this will— can be learned with, with Agents Flow from the outcome signal. And then the decision— every decision will be auditable. So per run, you can actually inspect it. And then you have access to that kind of inner loop because you influence that loop, how your Codex or Claude code— although I must admit I haven't tried it with Codex yet. I was purely doing all the experiments with Claude code. I did install it on a server in the cloud and I've run it for a week, different tasks, almost a whole week. night and day, so a lot of runs just to test it. And yeah, it worked. And yeah, this is the slide I was mentioning I'm using also several times because I just think it's just, yeah, showing exactly what kind of problems we are always facing. So if you're having that loop, someone still needs to configure that loop, right? So if you're doing loop engineering for your coding agents, Then you still, if you have several different models, you have different model efforts, all of that you are faced with. But okay, but if my coding task is now this and that, yeah, maybe I just set it the maximum effort, then the model will be doing the best and I'm using the best model, and then I will have the best outcome. And you won't like granularly adjust it for whatever the model needs to do for you. So this means you're faced with which model, which effort, skill, which role and verification passes, and so on. And this brings me directly to the problem with token maxing and token budgeting. So what you do is like what I said with setting just the most capable model. And to maximum. And I, like, I'm not no stranger to that. I tried this as well. And I was also like a little bit thinking, you know, maybe if it's— if I'm using the most capable model and then more effort, more thinking, maybe the results are generally better. So what we do is then you have a lot of thinking, but thinking tokens, all of that, and you spend a lot of time waiting because the model thinks longer, it takes longer. You see a token count going up. Then of course, what you have is you spend more money, you burn your spending much faster. If you have more restrictive API access or that your monthly rate is just like you need to wait like every 5 hours or whatever. So yeah, you get the gist, like this significantly slows you down. Either you have to wait because it thinks longer if you have like a high spending rate, or if you have a smaller spending rate, smaller token budgets, then it can be that you hit just for certain models like that 5-hour window, and then you have to wait and you can't fulfill the task in one. 1 session. So what then agent flows does, or the substrate in general, the policy does, it learns which task type earns which spend. So I tried this out like I said. So I had like for the task I had for the coding task I gave it, I had like the simple was haiku and then some sonnet and flow and checking haiku and like different— it routed it to different models and in different effort levels. And yeah, you see it, like I had 780 runs. And like I said, it took me almost a week to do all the tests and look at it from different angles. All right. So yeah, that's basically the gist. So Of course, setting every coding task to max effort, best model or skills, always verify, high retrace, that's actually not really a strategy, right? That's just talking bill and you will just increase the spend with that and burn your tokens down. So what's then happening is, so yeah, you have your user task, And it folds the signature. So it learns it, it looks it up. And don't mind the UCB, it's just for me being precise, but that's just how the policy— so it's a basic bandit reinforcement learning. So a very classical reinforcement learning algorithm. It learns and then it just maximizes the reward. The reward comes from different parts. So token spending, And accuracy, things like this. And I have 3 different judges which are wired to actually grade the coding results, and so that they can agree, they have very specific instructions how to do that, how to grade that, and that I get proper feedback. So this is also— you don't have to hand-implement anything of that, that works out of the box. And then It learns based on the task it has for your coding tasks for different repos. It learns actually how to handle those things. So then it takes the actions, and then Cloud Code just executes this configuration. And if it works properly, and it learns on the fly, so it constantly learns. So it's wired. It It updates, it gets judged, and then this is one of the objectives. So Belfort objective, how the reinforcement learning bandit learns it. And yeah, that's basically part of— so what it does, it explores. So it explores it, how you work on the fly, or you can also do this with— there is the SWE Agent. I'm sure some of you heard of SWE Agent, SWE Bench, and SWE Bench Verified. They have actually also in their repositories on GitHub, there's something called SWE Smith. S-W-E Smith. And what you can do with that is you can give it access to your repositories, previous pull requests, and just alter little things. So to introduce bugs, and this is— they did it to scale the learning for coding agents, but you can do the same in training than agents flow. So you just use an alter existing repository, then it can learn how you're working or your company is working with the code, what difficulty this is, what tasks you have, and then you can train the policy and then you can just use it on your coding task. Let me see, it's a little bit later. I show you that later. But basically what the good thing is, If you have learned that, and usually you have in, in your teams, how your teams are organized, of course you will have very similar tasks. So if you learned the, the policy one time, then you can transfer it to other tasks. And if things diverge, then Agent Flow learns it on the fly and will adjust. And yeah, here you see the the learned capabilities. So like I mentioned, there's the allocation per pipeline stage, and so it routed itself. So I didn't teach it anything.

It—

the policy learned it itself that, yeah, for coordination workload it used the Opus Medium, and for planning for a simple workload it used Haiku, and so on. So for different parts for heavier or not so heavy planning or building, it used the different models accordingly and learned it on the fly. Basically 2 patterns from those 3 signatures. What I just showed you is just a better overview. The simple, so factual lookup, single doc approach like things like this and then also refactor cross-file coherence and tested workload scales to different and more tokens and more difficult tasks it needed to do. An interesting thing here was really for checking that the Haiku actually did suffice, so just a cheap model. And yeah, this is what I mentioned with a warm policy. So you can actually adapt on the fly. And what I also tested is, so what happens if a new model comes out, right? So we have that always all the time. So they are having that model release race. And the question is really, is it really worth all the higher spend? So sometimes it's included in your plan for a short amount of time, then later you have to pay more or change your plan and things like this. What is possible, so you can actually test. If you have that testing phase where it is included or for lower spend in your token plan for your API for Cloud Code, then you can test it out, see if it's really worth it. This is what I did, and then it will adapt if it's worth it, it will learn, so the policy will learn. And route it if it's okay. And if it's not, then it will just ignore and use the still working, perfectly working models for that task, for the coding. Yeah. And this is here about the workload-appropriate Pareto. So it really is about spend more where it earns the quality. And this is what I tested. And this is how the setup works, so how the policy is evaluated. How the loop, kind of that outer loop, what agent flow does, what you can do with it gets evaluated. So it's really reward signal driven. And yeah, you see that it's cheaper for simple refactor, a little bit more for the coordination. So, but yeah, it goes for the better quality and still saving you a lot of tokens. Over half of the tokens get saved compared to if you just have it statically set. Another thing what I think is really good, what I like is that because you have that policy and it gets recorded, can access every decision and you can persist it server-side. So not just that, you know, all right, so this run did cost this and that, but you can actually look at it and you can see which stage burned it, which model was chosen, what the judge said. So like I said, I have 3 different judges wired, and there's also a reason why I did this. So I chose from different model providers. So that it doesn't have provider bias. And then I chose the different models and they, yeah, they agree on that. So, and you can actually see the reasoning and then you can audit it if you're interested in knowing about the code. And the same goes if you would apply it to your general multi-agent system without that that coding case, but I don't know, for customer service agent or something like this. And yeah, the auditable coordination layer is really what this framework makes the coordination also queryable. So you can really look that up. Yeah, for everyone who wants to try it out, Adapted, like I said, it's open source. I have all the links here. I'm still building since I'm alone. It takes some time. And I just told Angelina earlier that I had a terrible wasp invasion the last 2 days. So Wednesday, Thursday, and that set me also a little bit back. And yeah, I'm still having also like a job I'm working. So I'm consulting for companies. I'm actually implementing multi-agent systems myself. So today was a day with a lot of work. So if I have some of those days, of course, there's less time I can spend on improving that open source part right now. But yeah, just if you're interested in that, just keep looking at the repository. As I find time, I will update that. If you want to stay connected with me, here's the thing. And I will just show you a notebook that you can see this also a bit more live. And let me enlarge it a bit. All right. And yeah, you can access it via the LangGraph GitHub. So you saw the links in the slides. So you can just access that. And what you can do is then just you can run it. So basically run all. And then you see you get access to what I trained with, with Cloud Code on that server deployed. And yeah, if you go to the repository also, what I trained with, task it was and all of that. This is all transparent, so you can look it up how I actually validated all of it, what kind of task it was and all of that. So I'm 100% transparent here. And yeah, here what you see is— so this is just one substrate memory row, and every field, like I said, is really queryable. So ask for substrate Why did you pick this at this point at the reward table? You can basically do that. Here for the second section, remember I said the 2 patterns from 3 signatures. Here we have that the coordination's top 3 have all Opus at the plan. Substrate really learned that class needs capability at planning, then simple and refactor, top 3 are Haiku everywhere. And yeah, the, the 2 patterns really emerged from those 3 signals designed as different signatures from— and yeah, like I mentioned during the slides, really from, from the reward alone. And for the token budgeting versus the token maximizing So here the substrate spends a fraction of the token maxing baseline. So the baseline was basically the high Opus max effort and just have it fixed set. And yeah, you see here then the results for coordination. The substrate stays close on cost but the same quality. Because the reward signal said that the cheap arm doesn't hold there, and this is workload appropriate, like I mentioned on the slide. So where it actually earns to keep, then it will go for the higher effort and the better model. But if it doesn't, then it won't use it. And there, this is what you see with Plan-Opus-Max. And then it just used a cheaper model for building and checking. All right, and yeah, this is just for the 4th one. So yeah, free profiles, free policy. So like I mentioned, I had that 780 runs, same substrate, change the reward profile at import time, and then it flips and Yeah, coordination then picks the Opus Medium. Plan Premium picks Opus, and Simple picks then Heiko everywhere. So that is that you see, like, you can, like, also if a new model comes in, like I mentioned during the slides, then you can actually learn that it gets you the keeps. And yeah, this goes further for the policy adaptation. So when the action space is really expanded, so you can— it will flip. So coordination flipped to that opus marks because the reward earned it, like you see earlier above in the cells. So yeah, simple tested opus and rejected, and Yeah, it didn't give a proper reward back. So that was really like also for me, like a personal learning, like it makes sense to adjust this and actually also makes sense to adjust it on the fly. And yeah, that's really kind of, if you're working a lot with coding agents or Claude Code, Then it's really when a new model release, can I plug it in? And yeah, can the system decide or can Agent Flow decide? Yes, it actually showed that this is really possible. And like I said, you can also audit it. So this is, yeah, just it works with just— getting the access to that via the authorization layer. It's stored or it's done all via MCP and you can just get access to the JSON. That's about it. Any questions? Angelina, do we have any questions from anyone?

Not yet. We don't have any questions, but I have some comments. Maybe we can have a little bit discussion. So I think this is really interesting in the sense that basically the budgeting of tokens can be learned. I think it's a philosophical question and we don't want the developers and we ourselves don't actually want to pay attention to that. And the best is proven empirically. So we know that this task can be done towards the performance requirements that we ask for, and at the same time can minimize spending, then that's kind of effective already. But there's so many little decision points happening throughout the whole harness pipeline. It's very hard for us to pay attention to it. And it's not something that we can just write a policy and say that when you see this and then spend less money, and when you see that, that doesn't work. work, right? The best way is actually learned.

Yeah, yeah, it's, it's that you get it, give it out of, out of the hand, right? You don't have to, um, take care of it yourself because then you're busy in first setting it properly up and then you don't— it, it also is like it might not always be always, um, that it, it doesn't always stay the same, right? So maybe things change a little bit with the task you're doing, and then you need to revisit your coding harness. And if you have to do that, then, yeah, the policy from Agents Flow will adjust to that and adapt to that, adapt to new tasks. And like I said, with the new models and all of that.

Yeah. And I know that the mechanism you use to solve this problem is multi-armed bandit which is more commonly used in like A/B testing, right? And complex testing. How did you come up with this idea of using this mechanism to solve like, you know, the spending problem?

So it basically came from an earlier research from a few years back where I learned, well, I developed a deep learning architecture which already used attention. And there it learned what kind of— well, I had different experts, different expert smaller networks into that. So basically similar, more or less like agents, but it wasn't called that back then. So I had smaller expert networks and they were all in a complete system. And then the attention was deciding, does it need all experts? Does it need just a few experts? And that was the initial idea in, in using that. So since I can't get fully access to attention, so that was just like, all right, so I need something where I can fairly robustly learn the policy. And of course, I need reinforcement learning. So there are only so much classical reinforcement learning algorithms which can actually mimic what I did back then. So yeah.

Yeah. And I don't, you know, for effective token management, or, you know, for people who are budgeting for their tokens, I don't know if you've heard of this. This is an adjacent to what you're doing. So I'm seeing some companies selling tokens at fraction of OpenAI and Anthropic prices. And my initial read was some of them are just compute arbitrage from overseas, like China, right? And also they're building much cheaper models. at 10% of the price, like DeepSeek or QWEM models are really cheap, right? So how do you think about integrating those different models, cheaper models into agent flow? So the combination will be very powerful, right? A learned harness versus together with managed model selection.

I feel that, well, these are different things. So the problem is if you're using Claude Code, you can't just plug in a DeepSeek model, right?

That's true.

So if you're using Cursor, for instance, you could use a cheaper model. That would be possible. But I haven't tested it yet that you can actually steer their choice of a model behind it because that is fairly hidden. But Claude, they actually give you access to do the loop engineering. So you can actually do all of what I said. Depending on what IDE you're using, the one or the other will be helpful. I just know that a lot of people are using either Codex or Claude, most, most of the ones I'm in contact with anyway. So that just basically covers a lot of the problem. And of course, for neither you can actually plug in then some cheaper model. It's just not possible, right? So yeah.

Yeah.

You need to coordinate in the ecosystem you're having access to.

Yeah, depends on really what you're trying to make as well. Thank you so much, Nicole, for the session. But before we have the— if you want to connect with us, please, we have our QR code here. What's coming next? August 28th, we're going to be talking about this reinforcement learning part of agent loops. Many companies, many founders I've talked with are also talking about this self-improving side of things, self-improving AI agents, self-improving softwares. And then this becomes one of the most important topics that's in the industry nowadays. The next workshop, is also— the link is also here. We hope to see you next time as well. And let's thank Nicole, you know, again for today's session. As she said, she didn't forget about our workshop even though she was invaded by wasps and she and her little kitties were really, you know, injured. And we really appreciate, you know, we're still seeing her here being, you know, late in the evening on her side in Switzerland and with extreme heat together with wasp invasion. So we're very lucky to have you here today. Really appreciate it. And if you enjoy the session and you have questions, reach out to her. Sign up for the workshop. I'll share the deck and share the notebook. Also remember when you sign up, you know, opting for the newsletter so you can get all the materials. And we hope to see you next time. And if you enjoyed the session, one more thing, please support Nicole on her project on Asian Flow. I attached her link here in the, in the chat, so give her a star. I'll see you next time.

Okay.