Behind Navigator n2: How did Yutori make AI agents 10x cheaper? With Skanda Vaidyanath

Let me double-check if we're— I'm alive. Okay, I think we are alive. We're live now. All right, we can get started. Oops, sorry. All right, I can get started. Hello everyone, uh, I'm Angelina, founder and host of TwoSet AI Podcast. Thanks for tuning in today. Uh, some quick context if you're new here. So I started TwoSet AI as the notion of TwoSet, meaning 2 people talking about AI. And over the last year, I've interviewed roughly 100 founders and domain experts on the show. And every one of those conversations opened a door of a bigger engineering world behind them. And the engineers and researchers actually solving the real-world messy, novel problems the founders were describing to me. And I think that technical nuance actually deserves its own stage. And that's why I started this workshop series. The frontline engineers and researchers are the heroes here today. You know, my instructor Skanda is here today. I'll let him introduce himself to you guys shortly. But again, my goal with TwoSet.ai is to help builders, engineers, students, and anyone experimenting with AI like yourself.

Hi, everyone.

Like all of us, find your wedge the same way the founders I interview found theirs and build what you're passionate about and be positioned to sell it one day. And today we're going to— today we're going to go behind Navigator N2 from Yutori. Some of you may remember Yutori CEO Devi from the podcast. I'll share the link shortly in the chat. Today we'll get more in-depth on the engineering side of this model. Before I hand it over to our instructor, Skanda, today, a few quick housekeeping notes. So, so the talk will be around 30 to 40 minutes, and then we'll open it up to Q&A. Drop your question in the chat anytime, and I'll bring them in. One small ask. So if you find this useful, please subscribe to the channel. We really appreciate your support.

Thank you.

That's— you guys are the reason that keep me going. Skinner, I'll hand it over to you. Tell us a little bit about yourself and Yutori. Awesome.

Thank you so much, Angelina, for that introduction. Yeah, hello everyone, and thank you, Angelina, first for the opportunity, and thank you everyone for tuning in. My name is Kandar. I've been working at Yutori for the last year and a half or so, and we've been training models. I'm here today to talk specifically about our most recent model release. Super excited to talk about N2. Vittori has been training a whole bunch of browser use models of late, and N2 is our first computer use model in that series. N2 is a frontier computer use model, as Angelina mentioned. It's first place in 4 out of 5 industry benchmarks and a close second on OSWorld V2 as well. So today I'm gonna talk about how we train this model, and as a small team in a small startup on very, very small compute budgets and tight timelines, why we even decided to train such a model. So, um, yeah, hopefully today you'll walk away with some intuitions on how to train these models, if you should even train it if you're building a product. And, uh, yeah, we'll tell you about the design principles we had in mind and the algorithms we used. And there's a lot of details here in each of these slides. I've tried to keep it somewhat high level, but if there are any questions along the way, please feel free to interrupt and ask me at any time. With that, should we get started? Okay, so let's see. So before we start, I want to give a shout out to the team. So this is a team that built Entoo, uh, where Yutori It's a fairly small group and it was a couple of months of a lot of effort from all the people you can see here. So I wanted to give a shout out to them. So the name of our company, like I said, is Yutori. It's a Japanese word which means like mental spaciousness. And basically the idea is to give enough room in your life and in your head for you to do the things you care about and not the tedious things in your life. That was the company's mission. So in one line you can say automate the tedious parts of your life so you have space for what actually matters. And to do this, we wanted to start off by automating tasks on the web. So that's where we started training our browser use models and we built products on top of that. So some products like Scouts, Delegate, you might have heard of and tried. Scouts monitor things for you on the internet. Delegate goes beyond and is basically your own chief of staff. And then Yutori Local that can operate on your own local laptop. I was just talking to Angelina briefly before this and she was mentioning how she has so many tasks on the browser and web that she would like to automate. Yutori is exactly that and we build products and train models specifically to do exactly that. So before we get into how we trained Entoo and how we train our models, I wanted to talk a little bit about why we even considered training our own model. We started training our models a year and a half ago, so there's a little bit of historical context here on the needs that we had back then and why you should still maybe choose to train your models today. And to do this, I'll explain to you a little bit about what scouts actually need to do and why we need a browser use agent to do this, and then where computer use was back then, or browser use was back then, and where it is now. And why that gap in the landscape made us train the Navigator series. The Navigator series is just a series name we call for all the models we've trained so far. I'll tell you a little bit more about that in a second. I think it's worth calling out here maybe the difference between a browser use model and a computer use model. A browser use model is one that's specialized for the web and for the browser to operate on different websites. A computer use model is more general. It can operate on the browser and the web, but also on your— all the other local apps like PowerPoint and Excel and like Blender and all of that kind of stuff. So if you're ever confused in this, you can think of computer use models as a superset of browser use models. With that, let me get into what Scouts do and why we needed browser use agents to do this. So Scouts, the promise of Scouts was that they would monitor anything on the web for you. So you want a ticket booking, you're looking for apartments, you're looking for a restaurant reservation, it's like flight bookings, whatever you need. The point of Scouts was that it would monitor the open web for you anywhere. And then when something you want happens or when something you want is available, it books it for you or buys it for you. And that was the whole point of Scouts. So you don't need to stay on top of this. Scouts will stay on top of it in the background and you can just give it one standing instruction, walk away, and then Scouts will take care of this from that point on. So that was the promise of Scouts. Monitor anything for you on the web and check out and book things for you as well using your saved credentials, using your saved logins. So it needs to be, you need to be able to trust this model, especially when you're not looking. For example, you may say that there's a niche comedy show out in San Francisco that I'm trying to go to. Tickets sell out immediately, and I need you to monitor this website and then buy the tickets as soon as it's available. So that's one example. The problem with the web is that most of the web has no API where you can just call a tool to make these bookings or even get these, these bookings are available. So some websites, like for example Reddit, have paid APIs. Twitter has paid APIs. Google Search, you can do paid APIs with And you should when those options are available because it's more, it's faster and usually LLMs work better with these kind of tool calling APIs. But the problem is that most of the web doesn't have these kinds of APIs. So OpenTable, for example, there's no real API. Zillow, apartments.com, your favorite comedy show website, none of these websites actually have APIs. movie theater bookings. So you can't rely on building a product just using APIs to do this. And we needed actual browser use agents to go out on the web and not just gather these bookings, but also book them for you. And if you have a login, it's a login for you, then book for you. So we needed a model, a browser use model that was working underneath this entire Scout harness that was reliable, cheap and fast. It needs to be fast because many of these bookings are available for a limited amount of time, and you need to book them before they go out, or, uh, you need to book them before the price goes up or something like that. So that was why we needed a browser use agent in the first place. Now what about frontier models? At the time when we started, uh, computer use and browser use was not a very big thing. I think only a few of the frontier labs even offered them. Opus was probably the best at the time. And then OpenAI Operator was available, but it wasn't even available as an API at the time. So the— there were not many options for us to pick from in the first place. And back then, computer use models/browser use models were not as good as they are right now. They often used to get stuck on pages, they would make simple grounding errors, especially with like certain elements like date pickers. Like for example, when you're booking a flight, you— the dates come up, it would make mistakes on that. Or, uh, it would get stuck in these doom spirals on, for example, unfamiliar websites. It was not reliable to use on real websites, as a bottom line. Today, computer use has come a long, long way, not just from Frontier Labs but also from a lot of other people who are training smaller computer use models. A lot of other companies training other kinds of computer use models. It's quite reliable now. You can use computer use models on real websites and get tasks done. The problem is though that frontier models, which are the best by far, are too expensive to run all day. The promise of Scouts was that it would run all day for you, and whenever something surfaced, it would book it for you. So we needed something that would be cheap enough to run throughout the day. Compare this to a chatbot, which for example runs only when you put in a query or only when you ask a certain question, but scouts are asynchronous and always on. So we needed something that was a lot cheaper. And also frontier models, because of how many parameters they have and how big they are, they're often too slow. Even for like a simple login page or a simple form filling task, it can take several minutes. So back when we started training models, uh, the Frontier was not good enough. And right now the Frontier is too slow and expensive, but it's quite good, has to be said. Uh, so that's when we decided, almost a year and a half ago or so, to train N1, which is our first ever model we trained at Yutori. It was a pure browser use model designed just to operate on the web and operate through all these websites I mentioned and make bookings for you, collect like details for you from various websites and so on. So N1 was the first model we trained, and then in that series we then trained N1.5, which is a small addendum on top of that. And then now we have N2, which is the first full computer use model that we've trained at Jitoli. Uh, and I'm going to talk a little bit about N2 now and why it stands out in the entire computer use landscape. So this is a plot of various models' performance on an open-source benchmark called OSWorld V2. OSWorld V2 is perhaps the most challenging computer use benchmark out there right now. Every task is designed to take a human around an hour to 2 of work, so 2 hours of work per task. And these tasks can get quite complex, where one of them, for example, is to do a— to file a reimbursement for an overseas expense claim. So you have to find the reimbursement policy email to find which documents are required, then search your bank portal to find the relevant documents and the transactions, and then write up a supporting document in Word, and then fill up a reimbursement claim form on the expense portal. And you have to include the correct lines, the correct details, and then the form, as it goes on, asks for other information. This is, you can imagine, Many of us have done these kinds of complex tasks either for reimbursement filing or for filing for a visa maybe, or like applying for a job. These kinds of annoying tasks come up, I guess, all the time in our life and we would like computers to automate them. OSWorld V2 was the first benchmark that came up that had realistic tasks of these kinds that were as challenging. You can tell from the scores that the models, even the frontier models achieve on these tasks as well. So on the x-axis here is the amount of money it takes to do each of these tasks, and then on the y-axis is the— you can think of it as accuracy on this benchmark, basically. And as you can see, various Frontier Lab models are logged here. So we have Claude, Fable, Opus, Terra, etc. And then on the top left, you can see Navigator N2. Which is if you draw a horizontal line along the y-axis, you'll see that the next model that's offering similar level of performance is 10 times more expensive. And at the same price, it's over 3 times more accurate than the model along the same— if you draw a vertical line along the x-axis. So that was what we wanted to achieve with N2. A cheap model, a reliable model that is way more cheap, that's way more accurate than frontier models at the same price. N2 is a 27B model, which is why we're able to achieve this. The model is way, way smaller than any of the frontier lab models, which is why it's so much faster and cheaper. Across a whole bunch of other benchmarks as well, N2 stands out as state-of-the-art. on a lot of other compute use benchmarks that you might care about. I think, again, these are basically all the industry standard compute use benchmarks. And as you can see, N2 costs much less than the frontier models, $0.50 per million input tokens, $4 per million output tokens. And again, reminder that it's a 27B model, which is why we're able to offer it at this price. And it's so fast. So this is the whole pitch. This is what we set out to train, frontier-level computer use on all tasks that you might care about at a price that a product can carry. And that was what we were set out to do with Entoo. Maybe I'll stop there, and if there are any questions, I can maybe take that before I move on to actually training these models.

Okay. Yeah, I don't see any questions. Cool.

Yeah, so then I can move on to how we actually trained N2. So there's going to be a few parts in this entire section. The first part is the design principle that we adopted at Yitori to train N2, and then the data flywheel. For training any model, the data is the most important thing. And I'll talk to you about how we set up our data flywheel to work for our setting. And then I'll talk about rejection sampling and reinforcement learning, which are the 2 key algorithms we used in our setup to train these models. And I'll try to keep this somewhat, like I said, a little high level without going too much into implementation details. But if there are any questions, please feel free to ask. Okay, then let's get started. So, uh, this is perhaps the most important idea in this talk, or one of the 2 most important ideas in this talk. Uh, all the computer use models that were there before N2 felt like they were using computers in a human way. Humans use computers through the GUIs built for humans. So a human needs to click and type and scroll no matter what task they're doing. But a computer use model doesn't need to do that. The computer use model has access to tool calls. Like I said, if you have access to a Google Search API or a Reddit API to do a task, you should do it. It's often much faster and makes the trajectory less error-prone. Another modality that models are very good at, and we should take advantage of this, is coding. So if you want to build a slide deck, It's much faster to ask a model to code it than to ask it to click and type on the PowerPoint GUI. Humans often don't have access to these latter 2 modalities. We prefer to use the GUI for everything. But that's also useful because there are some interfaces where there's no other option apart from using the GUI. For example, browser use. Most of the websites you go on to aren't going to have an API and you can't solve it by coding. So you need to use the GUI to navigate through websites, for example OpenTable, to see what's available and book the right slot, maybe input credit card information and so on. So 3 modalities: GUI, coding, and tools. And it's— access to all these 3 modalities is fairly common. Like, you're probably thinking, I can— I've seen Claude use a GUI, I've seen Claude code, I've seen Claude use tools. Like, what's so special about All frontier models totally have these capabilities, but the hard part is to train them so that they know when to use each of these modalities and use them within the same task at different points at the same time. One of my colleagues put this very nicely in a blog they wrote, and he said that they can code, they can click, but they can't choose when to use each. All the benchmarks that I mentioned in the previous slide, they test these kinds of capabilities as well, and some of them even mentioned that frontier models often pick one of these modalities to use and then kind of get stuck and don't try other things in the trajectory and end up thrashing around and failing the trajectory or making it a lot longer than it has to be and a lot more expensive therefore than it has to be. The goal of N2 was to train a model that not only could use each of these 3 things one time at a task, but also through a task weave through these and use the right modality at the right time. To take advantage of the task structure and do it in the cheapest, fastest way possible for the end user. That was the design principle we had in mind to train N2, and that's what we wanted to achieve. Now here I'm going to play— I don't know if you can see this video. Yeah, you can see the video. So as you can see, this is an example of a real N2 session. Uh, it's from our— it's on the launch post if you want to see the video in more detail. But the task here is to take a scanned annual report and make it into a working spreadsheet. So imagine one of these like, you know, really long, like 50, 60-page PDF documents with scanned, uh, like tables in them and like brand spreadsheets and like numbers, and you want to convert that into an Excel spreadsheet that you can actually do analysis on. And that task is a really long task and it involves several steps. First, you got to read the scanned table off the screen because these are photographs, and you use GUI to do that. And then the model goes into coding and does the coding to see what it's able to see on the screen and then convert that into an Excel sheet. It's easier to do that via coding than it is to, for example, open an Excel sheet and then enter it number by number which is what a human might do. And then finally it switches back to the GUI to make sure that everything it entered and the analysis it did, all of that is what the user expected as well. So it does GUI initially, then goes to coding, then comes back to GUI. But it's even more finer than that because even while it's in that initial GUI phase, you can see it writing code to check certain things along the way, maybe check creating the file it needs to on the desktop and so on. So this is the whole idea. It's not just about picking the modality for the task, it's about picking the right modality at different stages of the task and then weaving through it so that it's accurate and also reliable and fast and cheap. With that design principle being set in our mind, the next most important thing to training any model is data. And this is probably the high-level— here's a high-level picture of the data flywheel and training pipeline of our algorithm, of our end-to-training. And as you can see, there are a few important stages here. And the first stage is task creation, so creating various tasks and verifiers to train the model. And then you need to stress test these tasks and verifiers and make sure that there are no false positives or false negatives. This becomes important later when we train our model because we don't want it to lead to any reward hacks or like benchmarking and things like that if something's gotten leaked. And then we go through this pipeline where this flywheel process where we train a model, then there's a failure mode analysis that we do to understand what this model is good at and what this model is not good at. And then that kind of influences what kind of data we create in the next round. So you can kind of see this flywheel process where every round we're curating and creating data that the model specifically needs. And then in over these loops, we kind of improve the model over and over, and eventually the model converges at this point where It has a high performance, and that's what we landed at with N2. The important part of this pipeline is that you'll notice that computer use agents are involved in every single stage of this pipeline. During task creation, we use coding agents to create tasks and verifiers, but coding agents cannot create these complex tasks and verifiers in a silo. They don't know what the GUIs you're operating with are, or what options and what applications they offer. So we use computer use agents to explore these various applications. So explore PowerPoint and tell me what I can do with it. Explore Excel, tell me what I can do with it. And we use those as seeds to then generate tasks and verifiers. And then during stress testing as well, we run a whole bunch of computer use trajectories on the tasks and verifiers we created to see, are we leaving any reward hack situations on the table? Are there any exploits that the model could exploit when we train on this. And there as well, we need computerized models to be really involved to do a lot of rollouts. This is extremely important for reinforcement learning, as we'll see later. And then there's the training stage itself, which is— I'll get to in a minute— which is the reinforcement learning and the rejection sampling pipeline. And for failure mode analysis as well, we have trajectories run by our trained model And then coding agents look at these to pull out the recurring gaps, the recurring mistakes, the pattern of mistakes that are being made, and we use that to influence the next round of generation. So you'll see here that it's not just creating tasks and verifiers in a silo, either with coding agents or with humans. The Competitus agents, preferably the one that you are training, has to be in the loop everywhere in this entire process. So the whole idea here is that Computer use models help generate better computer use data for themselves. And the reason this entire thing is— continues in a circle is that every stage your mistake patterns change, your model gets better, and you need to redo these stages at every point to kind of refresh your data and keep providing your model signal to learn from. Through this process at Yutori, over a few weeks, we were able to generate a lot of data across hundreds of applications, over 10,000 tasks and verifiers. that we could train our model on using RFT and RL. So this setting up this data pipeline initially, which took maybe a week or so, a couple of weeks, was very important for us. And then we could then bootstrap off of this over and over and run several rounds of this to get the final N2 model. Okay, so then moving on to the training pipeline. I'm going to zoom in on the training here. The training pipeline, if you kind of unfold it, looks something like this, where it's a repeated cycle of rejection sampling and reinforcement learning, which continues in several rounds until we reach N2. These 2 algorithms serve somewhat of a different purpose in the entire training pipeline. Rejection sampling is fast, it's less computer intensive, and it can help improve the pass-at-K of the model, which means it gives it some basic skills very quickly. The model may not be always reliable in doing these skills, but it knows how to do it nevertheless. Teaching these basic skills to the model can help it for the next RL stage as well. RL tends to be much more expensive and it's slower. Here we're just trying to consolidate the skills that the model has, make it more reliable, but then also teach it new skills via online exploration. So at the end of this, each of these rounds, after doing RFT, then RL, you should hopefully have a model that not just has acquired new skills but can do it reliably and is able to also adapt to any obstacles that come up online, which is where RL helps as well. And you can keep running these rounds over and over by just changing the data for each round, as we discussed in the data flywheel, and that should help lead to a better model at the end of this.

Process.

So that's kind of the high level of how the training pipeline looks like. Now let's dive a little deeper into rejection sampling. So this slide's a little heavy and there are several steps here. So like I said, the first step of this is doing error analysis to understand what your model is good at and what your model is not good at. Ideally, you want to focus on the stuff that your model is not good at and try to create more data for those. But also, you want to create some data for the stuff it's still good at because we don't want it to forget what it knows already. So we take our model, the best model that we've trained so far, uh, in our flywheel, and then understand its failure patterns on public benchmarks, our own internal evals, and we see what skills it's good at and what skills it's not good at. A coding agent is able to look at these trajectories and cluster these skills up into things like, for example, the model is good at here doc scripting, or the model is not good at slide formatting, or the model is good at spreadsheet charts and things like that. Based on these clusters that we formed, we then ask a separate batch of agents to author tasks and verifiers and environments for these The hard part is the environments. Like I said, you cannot create these kinds of the annual report tasks that I showed you in a silo. The agent needs to be able to see what's available on whatever VM you're running this on. Then as you roll out, how does the state proceed? What issues the model is running in as it goes through this task and use that as feedback to create a more robust task verifier environment tuple. And, uh, once we're able to do that and we have a robust set of tasks and verifiers and environments we care about that we're confident will address the gaps of the model that we would like to train, we then go ahead and collect data for this using our best model. This is where the rejection sampling pipeline comes in, because at every stage you're collecting models from the best collecting data from the best possible model that you have, and therefore the data quality keeps improving over the course of doing several loops of this pipeline. Now you collect the trajectories with your best model, and sometimes even your best model may not be able to solve some of these tasks. At that point, you can provide some small hints in the system prompt and to help the model just get over the finish line and push the task over. This helps in many cases, and we used it especially for certain cases where some Some skills are just very hard to do for the model without you giving it a hint. So we just give it that hint for it to unblock and then collect that trajectory. And once we train on that, the model will have the capability for the next round going forward anyway. An important step here in rejection sampling, because all these trajectories are gonna end up becoming data that you do for supervised fine-tuning, and there's a very high bar for the data quality that goes into supervised fine-tuning. So we can't just reuse all the trajectories that we've collected so far.

Far.

So what we do is have another batch, another round here of auditing all these trajectories for false positives and make sure that the data we collect does not have any trajectories that have been marked as a pass without it actually having been marked as having been a pass. This is important because all our verifiers are being written by coding agents and often in large batches. Like I said, we have on the order of 10,000 tasks and verifiers. So as we're dealing with this kind of volume, it's, it's better for us to make the data higher quality and lose out on some stuff than for us to admit data that's lower quality into our training pipeline. So this is why it's important for us to make sure, and because these verifiers are written by coding agents, it's very possible that they have bugs in them. So it's very useful for us to have this step where we kind of throw away some trajectories that we're not confident about and we don't want to go into training. And then we train from scratch. We have our base model and we train it with the new batch of data we've been collecting over the course of as many rejection sampling pipelines have been done so far. So if we're on round n of this pipeline, then we have data we've collected all through this entire round 1, 2, all the way to n. However, sometimes our model is constantly getting better over the course of this pipeline. As it keeps getting better, sometimes we throw away some older trajectories because they're not as high quality as we wanted. For example, some of the older trajectories may have these system prompt hints in them which we no longer want to use because it may add some bias in the way the model thinks. For example, the model may think in a way where it knows it has a hint available to it. We don't want those kind of trajectories. And we also don't want trajectories where maybe a weaker model that we had trained is looping around a lot, like it's, uh, not doing the task in the cleanest way possible. And we want the shortest, cleanest trajectory. So sometimes when your model has drifted far away from its initial state, it's better to kind of throw away the initial data and recollect all the data from scratch. And you just get generally a higher quality dataset by doing this, and it's a one-time cost you get to pay. So that was also quite useful for us. Any questions on the rejection sampling side of things? Okay. So like I said, the goal of rejection sampling is to teach the model some basic skills. It happens very fast. It's quite compute efficient. And we're trying to improve the pass at K of the model, so to speak, which means given a task, can it at least pass it once out of K attempts? And that's important because once it can do that, we can bootstrap off of that in the reinforcement learning stage. And that stage is where the model is really learning to consolidate these skills and learn new skills online as well. Then we have reinforcement learning. Like I said, rejection sampling increases the parser K, RL consolidates that into parser 1. It's also very important here for us to learn error recovery behaviors online because when we deploy the model after we've trained it, it's going to run into all sorts of trouble online. For example, pages may crash, the VM may not load correctly, applications may crash, like PowerPoint may crash on you or certain applications that maybe you expect to exist in that in a certain platform does not exist. Maybe you're trying to adapt from like, let's say, macOS to Windows or Windows to Linux or something like that, and certain things you expect to see are not where they are. There's a software update. So teaching the model how to recover from these kind of error modes is important, and that's where RL helps a lot as well. The pipeline for RL looks somewhat similar to rejection sampling, but the bar here is quite high on the verifiers. So as usual, we do the error analysis, this time of the rejection sampling model, to make sure that the RL skills that we want to impart are things that the model is still bad at or still struggling to do reliably. So we do error analysis and we ignore the skills that the model is already fairly good at. As usual, we create new tasks and verifiers and environments with computer use agents in the loop. To make sure that it's grounded in what the task can actually achieve. Then the third step here is very important where we roll out multiple competitive trajectories of the tasks and verifiers that we created to see if there are any verifier bugs that we've introduced and if there are any hacks that the model might be able to exploit online. As you probably have heard a lot, online on-policy training with RL or any other on-policy algorithms can lead to all sorts of hacks online if you're not careful about the verifiers you've written. At the end of the day, the model is just trying to produce a trajectory that passes the verifier. It's that simple. Your verifier has bugs in it and it's admitting solutions that you don't expect it to admit, or is being too strict and is not admitting solutions that it should be admitting. It can lead to training issues and your model ends up being not aligned or just not good enough. Typically what we do is that we run several rollouts of the tasks and verifiers. And then we try to keep these band of tasks and verifiers that are not too hard and/or not too easy. Not too easy ones are thrown away. The easy ones are thrown away because it's— the model already knows how to do it and we don't need to keep those tasks and verifiers around. It's a waste of compute. And the ones that are too hard are also thrown away because usually, depending on your RL algorithm, they are not going to be leading to training signal online. For example, if you use GRPO, the too easy and the too hard prompts don't contribute to gradients. Sometimes you might choose to keep the too hard ones in the hopes that the model will learn to generalize to it later, and we do see that behavior. But the real sweet spot of the data that you want to keep is that middle band where things are not too hard but not too easy, where the model has a lot of scope to improve its reliability and improve its error recovery behaviors on these tasks. We then go on and train with this dataset in a fully asynchronous GRPO setup, and that's our RL recipe. These are some quick implementation details of our RL pipeline. I don't want to get too much into detail here and I'm I think the— I'll just focus on maybe the important ones. Our entire recipe is fully asynchronous, which means that our rollouts happen asynchronously and our training happens asynchronously, both happening in parallel. This is very important for tasks like computer use because each task, as I mentioned earlier, can take an hour or 2 for a human to do. And if our GPU compute is just sitting idle while we collect those tasks, it's a complete waste of resources. So we parallelize these 2 and have rollouts happen in parallel to training so that, uh, we have high GPU throughput. So that's very important for tasks like computers. And then we have dynamic sampling, which essentially means that as the model gets better, we keep doing this filtering of too hard or too easy tasks online. We never want to train on samples that are too easy or too hard. So that's another thing that helps again keep your rollout throughput and GPU throughput high online. And the other thing that's important is to minimize the train-inference mismatch. I'm not going to get too much into this and make it technical, but this is a common problem that several practitioners have found while doing RL online. And using FP16 instead of BF16 can help minimize that mismatch a lot. The other things here on this slide are some minor points that we chose, and it made a difference for us, and we could see that these settings are better. For example, we have no reference model, and hence there's no KL to reference model. We don't have a discount factor. We have no access to process rewards. Uh, we don't normalize by response length, and we do training on the Miles framework. Uh, this is all stuff that you can decide for your own problem, but I think the first 3 that I mentioned— the fully asynchronous stuff, the dynamic sampling, and the FE16— might be free wins in your RL pipeline if you're training models using reinforcement learning. And finally, to conclude on what we built so far. So like I said, N2 is a 27B model on the computer use frontier, frontier level on OSWorld V2, OSWorld V1, and a whole bunch of standard public computer use benchmarks. The key ideas are 2 and 3, where try to do computer use the computer use way, which is not just the human way of clicking and typing and scrolling, Not just pure code or pure tool calling either. It should be able to use whichever interface is shortest and switching mid-task when needed. And build your own recursive data and training pipeline where computer use agents run at every single stage of that loop and make their own training data. As we get better models, we get better data, and you can keep continuing this loop until you saturate your evals. And I put up some links on the slides. You can try N2 on the playground at utopia.com/n2. And let us know what you think. We have also linked the blog and there's an API reference as well if you want to try the model in code instead of in the playground, that's fine too. Once again, thank you so much for everyone who tuned in and thank you, Angelina, for the opportunity to present. Yeah, it's my time. Thank you.

Hey, thank you so much, Skander. You actually got a bunch of questions. I'm going to show— this is a new feature I'm learning. I'm going to show this question. I think it's relevant to what you just said.

Yeah. Let me just read out. Any examples of a skill that were missing after RFT but emerged in RL? If you filter for the middle band, how do you ensure that the easy task reliability doesn't degrade? Yeah, good question. The main things that you can see that emerge after RL is, for example, in RFT we may have taught the model how to do, let's say, a certain kind of coding task, maybe it's like a bash coding task or something like that, the model is able to generalize that and generalize in RL to, let's say, like Python coding or heredoc scripting and things like that. So we may not have explicitly had RFT data for some of these scripting kind of tasks, but when you do it online with RL, the model is able to generalize and accumulate and kind of consolidate slightly orthogonal but similar skills as well. And if you filter for the middle band, how do you ensure easy task level doesn't degrade? This is one of the great things about reinforcement learning and on-policy learning that I think several people have spoken about. The rejection sampling stage has a higher propensity, I guess, to forget than the on-policy learning stages. So RL actually you can ignore the easy stuff and it just does a better job of retaining those skills and it doesn't forget as much as if you do RFT, which is why I'd mentioned earlier that in RFT at each pipeline, we don't just take the data that the model is not good at, but the entire data from all the pipelines so far because RFT tends to forget. But RL has this nice property because of the on policy learning or the reverse KL. I'll let you look into the literature on this, but it's known, it's a known property of RL that the forgetting properties of it is quite good. It doesn't forget much. So yeah.

Okay, I think we also, we have another relevant questions, but I'm gonna share the last page. So just in case people wanna sign up for our new workshop for the next week, and I'll show the next question. Any frameworks you are using for asynchronous RL? If not, any suggestions for a battle-tested framework?

Yeah, so we use the Miles framework. It's an open-source framework that supports asynchronous RL. We also used to use VRL before this. VRL we moved away because it was not being maintained as well, but that should also have asynchronous RL, I believe. TRL is a very common one that people use. I don't know if TRL supports asynchronous. MILES is fairly friendly. I'd say maybe start with MILES.

Fantastic. I hope it answered the question. We have some other questions. I'll start from the beginning. Okay. So from Philip Kerger, has there been any discussion at Eutroia on open weighting the model?

Yeah, good question. I don't know if I have authorization to answer this, but we have discussed it. We'll maybe open source it soon, or maybe— yeah, we've discussed it, but we have no decisions made, unfortunately. But yeah, 27B, I don't know if you can run it locally. I don't know what you have locally, but maybe if you have a beefy Mac Mini or something, maybe you can run it locally.

Yeah. Awesome. This one, agent got stuck on some React app. Yeah, some practitioners actually.

Yeah, good question. So N2 has— I should maybe mention this more clearly. N2 can, depending on the task, it can operate based on screenshots or based on the underlying DOM of the website, so, or the React app in this case. So, uh, I'm not sure what kind of agent exactly you're using. Is it a deterministic agent or is it, uh, a computerized model? I guess my response would be, uh, try it on the playground on N2 and see how it does it. But, uh, the technical answer to this is that N2 has access to both the screenshots and the DOM, so it can choose whichever one it needs to at each point. And Again, the error recovery if something goes wrong is helpful as well.

Awesome. This one, can you talk about the OSWorld-style benchmark and what other factors you need to consider to relate it to trustworthiness of agents?

Yeah, so the OSWorld benchmark is, it's a bunch of computer use tasks. Like I said, the one example I gave was the reimbursement filing thing. There are some other examples like, go online, find a list of all the restaurants in this area, and then create like an Excel sheet for me. It's like your classic computer use kind of things that involve multiple apps, involve a little bit of coding, involves UI and things like that. And what are the factors you need to consider related to trustworthiness of agents? I'm not sure if this question is from the Point of view of deploying your agents, or I don't know if you're asking about training verifiers, but I think one thing I guess we're looking for in trustworthiness is, is your agent doing reasonable things given the task? Is it choosing the right modality, for example? Or, for example, just choosing the wrong modality can make it go into this really windy path that that makes it do things that you're kind of uncomfortable. Uh, if it's clicking around a lot, typing around a lot, and where you know that you would have just done it in a much simpler way, that kind of stuff can make you feel, I guess, somewhat uncomfortable. Uh, I don't know specifically that these benchmarks are testing for that. It's kind of just saying that if a model does well on OSWorld, then it probably has some reasonable judgment and how to do computer use tasks, and hence you should trust this model. Um, yeah, I guess I need an example of how you mean trustworthy to answer this more specifically. If, if you're in the chat, yeah, you can.

Yeah, uh, uh, Samaritan224, if you have follow-up, please type in the chat. Uh, he has a— they have another question. Is there a common thread between the tasks where the model training requires a human in the loop to nudge it in the right direction?

Yeah, good question. Most of these tasks are designed to be done end-to-end. So a human gives an instruction and then the human— sorry, the model just goes and does it end-to-end. However, like I said, we've been integrating our model into various products and one of the products is one of the things we try is to try to get the model to work on your local computer. And as you do that, you are still working on your computer and you're moving things around. So even if you're not trying to nudge it in the right direction, maybe you're just doing whatever work you're doing, the model is surprisingly good at adapting to your nudges and also just you doing your own thing. So this is where I feel like Training on these OSWorld-style tasks gives it reasonable judgment on these kinds of computer use tasks, and it knows how to recover. For example, in that— I keep coming back to the recovery thing. Computer tasks would be super easy if whatever you wanted to do just happened magically. This almost never happens. Like you said, clicks can misfire, you get some random popup, maybe the human does something to interrupt the task or maybe push the model on the wrong side of things, and it's able to recover from all these things, which I think is quite—

Cool. Thank you. Uh, we have one more question. This is also my question, you know, users, ah, yes, you know, do stuff for you, but the sites keep adding guards against these bots.

Yeah, yeah, yeah, good question. Now, uh, like I said, um, where does this end? I'm not sure. I'm not, I'm not gonna make a prediction on that, but I was just telling Angelina that, uh, we use both in our products. We use APIs when they're available. They're faster and LLMs tend to work better with them when they're available. Like you said, browser use especially, computer use I think is mostly fine because it's acting on a VM, but browser use on these open websites can be tricky. It can be going to work against all these bot detection and CAPTCHAs and things like that. Yeah, I'm not sure where it's going to end, but I feel like there are more and more products coming out with these kinds of capabilities and offering these services. So I'm assuming that the entire ecosystem is moving towards that. For example, I know that there are some payment platforms also that support like agentic payments and things like that. So looks like the ecosystem is moving towards there, which is cool.

So this is really interesting. So it means that when you're building a a product or you're going to build a product, you should think about having not just build it for the humans, but also build it for the agents.

Absolutely. Yeah. And I think there are lots of interesting questions here, especially with like, you know, booking and buying things for you. Like if it buys something wrong, I guess, who is responsible? Are you responsible? The person who made the agent responsible? Or like, is the payment company responsible? I'm not an expert on this, but apparently people are figuring things out. So that's cool to see. Yeah.

Who do you think should be responsible?

Oh.

Any of our audience, please chime in. Like, who do you think should pay for that?

I'm curious what your audience thinks, for sure.

I'm curious too. Yeah.

Okay.

And yourself? You said— did you say who do you think should be responsible for? Me?

I haven't thought about it that much, but it does feel like the— if the— this can get into very gray areas. It's like, if the instruction was not clear, then can you blame the agent? Or if the instruction was clear, then I think it's quite obvious to say, okay, the agent clearly made a mistake and did something wrong. But if they— often the problem is with these kind of like personal agents, users expect that the agents know something about them that they maybe haven't mentioned explicitly. And in those kinds of scenarios, I guess, who takes the blame? I'm not sure. Yeah.

I don't know. If we think of it as a human being, right, if we do have a personal real human being assistant, then it's maybe similar. You kind of have to be responsible and maybe you have to have some guardrails in place. Like if it's more than $500, and then I don't want to, I don't know, depending on the risk tolerance.

Yeah.

Or maybe insurance company at some point will come in the landscape.

Perhaps.

Right. Do you think— I have one more question for you. Do you think landing pages are dead? Because we are all searching within AI, right? And we're going to have our agent to touch other, you know, the providers' landing pages. Do you say— oh, I should say this. Are landing pages for human beings dead? Are they?

It does feel like it's going that way, doesn't it? I mean, I haven't landed on many landing pages in a while. I'm kind of just going through ChatGPT for most of the things I want, or I have these AI agents book things for me, buy things for me, and things like that. So yeah, insurance against personal agents sounds like a business that could work. Yeah.

Yeah, sounds good. That's all really good questions. Thank you everybody joining in and thank you, Skanda, as well for this amazing demonstration.

Thank you so much.

Yeah, I want to mention one more time, if you want to try out Intu, this is the website so you can just go check out their playgrounds. I think I need it because I want to automate the creation of my StreamYard workshop events. Again, next Friday, October 2nd at 10:00 AM, We have Joseph Enochs. He's the Chief AI Officer at Enterprise Vision Technologies, and he's also an O'Reilly author on video generation. So their company worked with many of the Fortune 100 companies and helped them adopt AI. So if you're interested in how enterprises actually get AI from the boardroom into production, you know, this is the one you should come. So I hope to see you all there next week as well. And if you or your team is working on something engineering heavy that deserves a bigger stage, DM me on LinkedIn or just send me a note. Happy to host you for a workshop as well. And my shameless plug one more time, subscribe to the channel, help me keep on going. Thank you so much. Thank you, Skena, as well. See you all next time.

Thank you.