OpenAI's New OPERATOR Agent Surprises Everyone!

So OpenAI have just gone ahead and released their first AI agent. And this is truly a game changing product as it sets the benchmark for what will be a glorious future of many AI agents. Doing various tasks for us in the background while we get on with real productive work. I think this is going to completely change the game and there are many things that most people did miss. So let's dive into the first demo and then I'm going to explain to you guys more about this AI agent.

Okay, so this is the Operator homepage. It lives@operator.chatgpt.com It'll be accessible as soon as the live stream is over. And as you can see, the interface is very similar to ChatGPT. You can type in a prompt and an operator will try to execute the task to the best of its capability. You'll also see we have a list of pre filled prompts here. These are not really meant to be recommendations. These are meant to be things that to give you an idea of what Operator can do, we have also collaborated with various brands like OpenTable, Allrecipes, Type, Subhub, Uber, Thumbtack, DoorDash, eBay, Target to make sure Operator really works well on these websites. But also we think users will find Operator very valuable in interacting with these platforms. So with that, let's jump in with a demo. Okay, so I'm going to start with something fairly simple. I'm going to use Open Table and say hook me a table for two at Beretta tonight at 7pm okay, so.

You specifically chose OpenTable?

Yeah, in this case I'm asking operator to use OpenTable to book a table for two at Beretta. Beretta is a restaurant in San Francisco. It's great, you should try it out. And at 7pm and I'm using OpenTable in this case. But I could have easily said just do Beretta and it would have probably gone to search engine, figured out how to make a reservation as well. But let's see what it does.

So can you explain what's happening in this?

Yeah.

Right. So I'm going to expand this a little bit. So as soon as I typed in the query, Operator instantiated a completely remote browser. This browser is running in the cloud somewhere and as you can see, it's already up and running. My hands are off the keyboard. I'm not typing these things.

This is just the AI is clicking around.

AI just clicking around. It started this browser session. It knew where Open Table website is, which is OpenTable.com as you can see it's summarized chain of thought here as well, which is it's gone to the URL searched for Beretta and something cool really happened, which is for Some reason operator OpenTable thought we were in Virginia and it autocorrected itself to San Francisco. This is using. So like ChatGPT in operator, you can also give custom instructions. So I'm going to show this really quickly here just to. Okay, so I've given a custom instruction that for queries that need it, I live in San Francisco. So Operator recognized that and then autocorrected itself to go to Beretta. Okay, looks like 7pm isn't available. But you know what, 7:45 is just fine. So we're going to go do that. So in this case, Operator came back and this is a really good example of task delegation where Operator needs help or needs assistance or just wants to ask you something, he'll just come back and you answer that.

So in practice you wouldn't have had to watch this.

You could have just let it go.

Off while you're doing other things. Then it would come back and say, hey, I can't do seven.

Totally.

Five.

Yeah. And we're starting with a web app. You'll get notifications, etc. When operator moves into mobile, you'll get mobile notifications, much like interactions we do with general apps. Okay, yes, that's great. Let's do it. Okay, so again, very simple interaction as you would have with an assistant, which is, hey, I found reservation 7pm Wasn't available. Let's do 7:45. And again, you can see Operator at this point, I said, okay, should I. Again, this is a really good example of the confirmations work we're going to talk about a little bit later. But you know, before doing an action, which is sort of irreversible in this case, we can cancel iteration, obviously. But again, taking a critical action. Operator is asking us before actually doing it. In this case, I'm going to say, let's do it. Okay, it was pretty quick. I would say like, you know, 50 seconds.

So right there we got to see the first case of the OpenAI operator agent. Now, what's crazy about the operator agent is that this is something that is completely autonomous, meaning that you can literally give it a task and you don't actually need to be in the web browser monitoring the agent. And I think that that is an important feature because most agents that we're going to have in the future, and most ones are actually useful, are ones that we can say, you know, what go off and do this whilst we do something else. And that is of course, where the leverage is. And with this agent, one thing in particular that I did find was that at certain checkpoints, it did actually ask you for more permission, which is of course really good. But I do know that in future it's quite likely that these models will be able to reason by themselves and figure out how much autonomy you want to give them. Now let's take a look at what happens when you actually want to stop Operator in the midst of doing something and actually intervene with the task, why this is important and how you should actually do that.

I can chew at this point and I'm going to just click this button called Take Control. So this remote, as we were talking about, like Operator, fires up this remote browser to do it. We almost think of it as Surface area where Operator can work and then I can work. For example, in this case, I took over Control from Operator, which is also key to sort of how we think about user and user controls. At any point in time, a user should be able to take control and give Operator instructions or tell a little bit more, guide a little bit more, et cetera.

Passing the laptop back and forth. Just like you did with Ray.

Totally, totally. Exactly right. Just like in this case, I'm going to make those two and then I'm just going to tell Operator. This is again, very much like if you and I were working, like, hey, I did this, can you fix this? And I'm going to tell Operator I added another egg. Good to place order.

Now, can operators see what you're doing during takeover mode?

Great point. So when you take over, it's very much just like a session with your local browser. It's completely private. Operator cannot see. And this is one of the. Part of the reasons why I have to tell Operator. You don't really have to. It can look at the last screenshot and try to guess it, but it's really good. It's sort of like if you and I were working together, I went off and did something and I come back and like Ray, I completely messed it up. Can you fix this? And I have to tell you that. So in this case, I'm going to tell Operator. Go ahead. And now I'm passing back the control to Operator. It's a completely private session when you take our control. This is also the. You'll notice that I'm logged into Instacart here. I did it before the demo or has been logged in for a while now. And it's Again, very much like your local browser, when you log into Instacart, until the cookies are cleared, you stay logged in. And we have really good controls. You can go in settings and control and remove at any point in time.

Now, what we actually need to take a look at is how the Operator Agent truly works. Now, most people don't realize that this is an agent that is actually just like a human. So it looks at the screen, it analyzes the pixels, and then it reasons about where it should click next and what the next action it should take based on the end goal and the prior steps taken. So the reason that I think this is really important is that this means that this kind of tool in the future is going to be able to generalize across a wide range of different tasks you can, meaning that, you know, these simple tasks that we're now doing in the browser are going to become a lot more complex as time goes on and as the reliability of the model does increase.

Let me talk a little about the research behind it. So Operator is based on a new model we've trained at OpenAI, which we're calling the Computer Using Agent, or CUA for short. So CUA is a model built off of GPT4O, but it's also trained to use and control the computer in the same way that humans can, by just looking at the screen and using a mouse and keyboard to control it. Before, if you wanted to build something like operator without CUA, you'd need to use some specialized APIs. For example, if you wanted your model to buy stuff from Instacart, you'd need to figure out if instacart had an API. You'd need to figure out if that API had all the functions that it needed. And you need to give your model the specs of that API. But if your site, like most other websites, did not have an API, then you're out of luck.

This is just using screenshots, no API, nothing, just more API.

Yes, and that's where KUA comes in. By teaching a model how to use the same basic interface that we use on a daily basis, it just unlocks a whole new range of software that can use that was previously inaccessible.

And so this is keyboard and mouse, Right? It's kind of using keyboard and mouse, just like.

Yes. And that's really what the cool research project is about. It's about removing one more bottleneck in our path towards AGI.

So you see right there, they actually spoke about how that is, of course, one of the bottlenecks towards AGI and they are trying to remove that now. This is a system that is actually quite good at browsing the Internet, but it isn't as good as humans just yet. And you might be wondering, okay, well, if that is the case, where do I stand in terms of maybe my job security, how good it can use computer compared to the average actual human? So they actually did a benchmark where they pitted operator agent against humans. We can actually see the results of these tests and we can see exactly where humans are in comparison to these models. And I think it's going to be super interesting to see how these benchmarks change over time and how these models manage to rapidly get better.

That said, we can look at a few benchmarks and kind of quantify how good operator is right now. So one of the first benchmarks that we're going to look at is called osworld. Osworld is an eval that measures how well AI agents navigate common operating systems like Linux. On this task, CUA gets a 38.1% score, which is higher than other publicly published results. Human performance in this task is 72.4%. So we still have room to grow, definitely. The other eval we'll take a look at is called Webarena. Webarena is an eval that measures how well AI agents navigate some common websites, like e commerce websites or social forum websites. So on this task, Kua gets 58.1%, again, higher than other publicly published results, but still falls short of human performance.

Still a way to go.

Still a way to go, yes. One thing that's important to remember about Web arena is that even though it's the web, we're still just giving it the same universal interface of screen, mouse and keyboard. We're not giving it any extra information that might help it do the task, like the raw text of the webpage or information about which buttons are clickable. And all the information it needs, just like humans, is just in the screenshot.

Now, another thing that they will have to focus on is, of course, safety. Using an AI agent opens up a variety of different possibilities. Risk and security vulnerabilities. Imagine you're having an AI agent or even a bunch of AI agents do many different things on the Internet that are borderline illegal harassment or just a variety of different tasks that it really shouldn't be doing. They've developed essentially a kind of framework that ensures the AI agent stays aligned, ensure it doesn't mess up, and of course, ensure your personal details aren't leaked or the agent doesn't do something that compromises your security. Yeah.

So I think we're all very excited about this vision of operator doing your chores for you. But it is one of the first agents that we're putting out in the world which has real world side effects. And so we thought carefully about how to deploy this safely. The framework we used to think about this was one centered around, so, for example, what if the user is misaligned? So maybe they're asking for a harmful task like buy a weapon or something like that. In that case, fortunately, we've done a lot of work with ChatGPT, so to bring over a lot of the same mitigations. So for example, we refuse harmful tasks, including harmful agentic tasks. We have moderation models, we have post hoc detection, we have blocked websites. And I'm kind of rattling off these mitigations, but that's really how we think about it. It's this stack of mitigations that each incrementally reduce the risk to the point where we feel comfortable deploying.

So all the confirmations that we're saying, hey, do you want to reserve the restaurant?

Yeah, exactly.

Should you buy the tickets? Those are all examples of the same.

Exactly. And I'm about to talk about the confirmations. So another area of misalignment is if the agent is misaligned. So if the model makes a mistake, maybe purchases the wrong item or yeah, books the wrong hotel room for this. Our main mitigation is confirmations. So the operator will come back if it's about to do something stateful and ask you so you can double check what its details in case it made some error. The third area of misalignment is if the website is misaligned. So maybe the website is fraudulent or it's a fake website, or maybe it's literally like operator, please wire me $100. We obviously don't want to follow those instructions. So we've developed our model to try to avoid those instructions and not follow them. But if that fails, we also have a separate layer on top. This is what we call the prompt injection monitor. Think of it as like antivirus that kind of observes and watches your trajectory and sees if there's anything suspicious. If it does, then it pauses it. So we feel pretty comfortable with our approach. But obviously safety is an ongoing process. We can't predict everything. So we hope to learn a lot from this deployment and iterate on our mitigations as we go.

Now, of course, that is remarkably important. We wouldn't want our agents doing anything that make them use our credit card and send others money. Imagine waking up to find out your AI agent has sent off $1,000 to a random scammer online. That would be something that's incredibly frustrating. Now, of course, you might be thinking, okay, the AI agent is here. We can use it. It's got a virtual browser and it's able to do a variety of different tasks that are quite useful to me. What about the future of AI agents, though? Are they going to be doing anything in the future? Well, of course. Sam Altman has actually spoke in this video, in this live stream about how in the future there's a lot more to come from AI agents. So it's. It seems that this is basically just their beta entry into AI agents. This is going to be something that's really, really simple so that people can truly grasp and understand the usefulness of these AI agents and of course, develop workflows that actually make sense. It's quite likely we're going to get AI agents that are a lot more comprehensive and of course, AI agents that specialize in other domains. Perhaps a coding agent. This is something that OpenAI have referred to very recently. So I wouldn't be surprised if in the future they actually talk about the coding agent that they use internally. It's definitely going to be an exciting time to be in AI.

This is really the beginning of this product. This is the beginning of our step into agents, Level three on our sys, on our tiers, and we can't wait to see how people are going to use this and to kind of work with us to figure out where exactly it should go. So, again, congrats. Hope you enjoy it. Thank you very much.