Bigger than Open AI o1 - Claude 3.5 Agentic Computer Use

The AI space is on a roll right now. I mean, the game does not stop. There is so much news that was announced just today, I can't even fit it all in one video. In fact, I'm only going to be talking about one particular thing that happened today in this video. But be on the lookout because I want to cover all of it at some point. I'm doing a larger AI news roundup. But yeah, there was like a ton of stuff that was announced just today and even more in the past few days. I mean, it is absurd, but today I wanted to focus on this. Anthropic AI takes a big whack at OpenAI introducing computer use. New Claude 3.5 sonnet and Claude 3.5 haiku. Obviously, the big thing here is computer use. What does that even mean? Well, I'll be blunt with you guys. They trained their brand new 3.5 models to actually use computers like a human can, and it is scary good. I have the demo set up and working today, so we're gonna check it out. They have limited the demo quite a lot, but obviously more expanded access is coming in the future. Computer use is still in beta, but it is absolutely the most advanced and impressive computer use I've ever seen from any AI, point blank, period. So we'll start off on their blog post here. Yeah, the first thing is that they upgraded 3.5 Sonnet, which is their previous flagship model and competes with GPT4. Doesn't exactly compete with the OpenAI 01 models, but Anthropic is claiming here that OpenAI's new 01 models shouldn't be benchmarked against Claude 3.5 Sonnet, the new version. They've also got a new small, lightweight and fast model called 3.5 Haiku. Basically, the TLDR here is that 3.5 Sonic got, you know, a pretty decent boost across the board, improvements over its predecessor. The most significant gains here you're going to see are in coding, which is pretty exciting. Although I have to say, OpenAI's new 01 models are really good at coding, so I'd be interested to see how it stacks up against that model. In terms of coding, 3.5 haiku is pretty darn impressive because it actually matches the old performance of their previous Big Boy model, Claude three Opus. So you're basically getting the same performance for way cheaper and way faster with 3.5 haiku. So let's talk about computer use. It is in public beta, meaning anyone can access it. It's a little bit tricky to Set up the demo yourself, though. I'm going to talk a little bit about how I set it up, but if you want a larger tutorial, please ask me. I'm not going to be covering that in this video today. It's only available through the API and that's how you get access to it. And developers or anyone can direct Claude to use computers in the way that people do by looking at the screen, moving a cursor, clicking buttons and typing text. This is their first frontier AI model to offer computer in public beta. And at this stage it's still very experimental. It might be cumbersome or error prone. It's not perfect, but as I said, there are similar computer use models out there, which I will talk about to provide context. They are telling us to expect the capabilities of this to rapidly improve over time, which is pretty terrifying. This thing's going to be able to do some things that we have never seen before. Various companies have already been exploring and implementing apparently the new computer use and testing it out themselves. But they do have a nice demo video and this should kind of give you guys a decent idea of what to expect from this thing. Prepare for your minds to be freaking blown. I really hope this music is not copyrighted. Anywho, let's take a peek.

I'm one of the researchers here at Anthropic Compute. Use is something that we felt was going to be important for a while now. And so today we're going to be talking about a very early version we have of computer use, walking through a representative example of the things we think it's going to be useful for.

Automating operations. Let me read this real quick to you guys. Claude is generating all of the computer actions shown here. This demonstration was recorded in a controlled environment with some supporting infrastructure to simplify and highlight the core capabilities of computer use. And this isn't the most impressive demo, by the way, that they have shown off. We're gonna take a look at those as well. But this is a nice, simple intro to computer use.

We're going to be going through a quick demo today. In this fictional demo, a customer, in this case the Ant Equipment company, has come to us and asked us to fill out a vendor request form. The data I need to fill out this form is scattered in various places on my computer. What we're going to do is ask Claude to look at the spreadsheet.

This is the prompt we're giving. Claude, fill out the vendor request form for Ant Equipment Co. Using data from either the vendor spreadsheet or search portal tabs. Very, very specific. Basically, this thing's just looking, gathering data and then filling out a form. Like I said, simple demo, but not the most impressive one that they showed off.

Check if Ant Equipment is in there and if not, move over to the CRM and try and find some more information there. Once it has this data, Claude's going to then fill out the form for us and hopefully transfer the information across to the vendor form. The first thing that's going to happen is Claude's going to start taking screenshots of my screen and quickly realizes that the Ant Equipment company isn't actually in the spreadsheet. So the first thing it does is it swaps over to a CRM and searches for the company we're interested in.

Pretty cool, pretty cool.

Luckily, we get a search match and Claude then starts scrolling through the page looking for all the information it needs to fill out this form.

It takes a little bit of time.

Transferring the information across without me having to do anything and goes through the steps and fills out all the information needed. And then submits the form. This example success.

It was able to submit a form and gather all the information just by clicking around on the screen. Pretty darn cool. It definitely knows how to use a computer in a very basic way. Right? But like I said, they have way more impressive demos. Now, before we look at those other demos, today's video does have a sponsor, which is pretty awesome. Sponsors literally allow me to do this full time. So huge thanks to you guys for sticking through them. Quick word from our sponsor and we'll be right back. If you're an up and coming content creator and already using tools like ChatGPT or Claude, you understand the potential power of AI technology. But today's sponsor SEO writing goes further. Being able to generate full blog posts with all of the fixings. It's your own AI partner for writing, formatting, scheduling and optimizing your content for the larger web compatibility space. And as a creator, believe me, I get it. Speed is everything. So check this out. Just one click and you can generate entire blog posts, complete with AI images and even relevant YouTube videos. Whether you're looking to produce a standout post or up to a hundred articles in bulk, SEO writing has you covered. But that's not all. You can also schedule and auto post your content, keeping it fresh with minimal effort. They also have a humanized text feature which allows your AI generated content to vastly stand out in comparison to others. Because SEO writing does all of that pre prompting work for you, it's also got deep web research with built in citations so you know that your posts are always accurate and up to date. And to top it all off, it is SEO writing. After all, their internal and external linking features boost SEO. And here's the best part, just for Matt Vid Pro AI viewers, you get a 25% discount. All you gotta do is use code MATT25 at checkout. So click the link down in the description to get started. Huge thanks to SEO writing for sponsor Today's video. Ready to supercharge your blog? Give it a try. Now back to your regularly scheduled content. Welcome back everybody. Let's continue exploring this insanity. Next up, let's look at computer use for coding.

I lead developer relations at Anthropic and today I'm going to be showing you a coding task with computer use.

And again, we have this same exact message.

So we're going to be showing Claude doing a website coding task by actually controlling my laptop. But before we start coding, we need an actual website for Claude to make changes to. So let's ask Claude to navigate to Claude AI within my Chrome browser and ask Claude within Claude AI to create a fun 90s themed personal homepage for itself.

So we're right off the bat already prompting Claude to go to Claude AI website itself and then essentially prompt another version of itself to create the website at hand. Which is pretty darn cool. So you'll see here goes off navigates.

Searches for Claude AI.

Absolutely insane. I'm gonna be so mad if this music is all copyrighted. I might just mute it right off the bat. But you can see here, it's literally prompting Claude itself in the website to build the website. Claude then goes through and builds that same website. There it is. Yay.

I want to make a few changes to the website locally on my own computer. Let's ask Claude to download the file and then open it up in VS code. Claude clicks the Save to file button, opens up VS code.

Pretty cool.

And then finds the file within my downloads folder.

That's insane to me. Already it went into the downloads folder itself. It's like, oh, there it is. We'll just pluck that out, put it in VS code or as I like to call it, versus code.

Perfect. Now that file is up and running. Let's ask Claude to start up a server so that we can actually view the file within our browser.

I wouldn't know how to do this by the way.

Claude opens up the VS code terminal and tries to start a server.

But.

It hits an error. We don't actually have Python installed on our machine. But that's all right because Claude realizes this by looking at the terminal output and then tries again with Python 3, which we do have installed on our machine. That works. So now the server's up and running. Now that we so it was able.

To find an error and then solve it all, albeit it's kind of like not a huge error. For example, if you didn't have Python installed at all, it would have to go through and then download and install Python. But I'm pretty sure it could also do that if the local server started.

We can go manually. Take a look at the website within the browser and it looks pretty good, but I noticed that there's actually an error in the terminal output and we also have this missing file icon at the top here. Let's ask Claude to identify this error and then fix it within the file Claude.

This is where it gets cool.

Reads the terminal output and then opens up the Find and Replace tool in VS code to find the line that's throwing the actual error. In this case, we just ask Claude to get rid of the error entirely. So it will just delete the whole line. Then Claude will save the file and automatically rerun the website. So now that the error is gone, let's go take a final look at our website and we can see that the file icon has disappeared and the error is gone as well.

Pretty good. Simple website coding, but still, like a lot of people wouldn't know how to do that. And it was able to just do it automatically, autonomously controlling your computer. So insane. And finally, they also have a demo for orchestrating tasks.

I'm a researcher at Anthropic.

I swear to God if this music is copyrighted, I am going to be so upset.

I'm going to show you a simple example of computer use today. My friend's coming to San Francisco next week and I want to take him to do some touristy stuff. I think doing a sunrise hike with a view of the Golden Gate Bridge never gets old. So I'll ask Claude to figure out some logistics for us. I'll ask Claude to find a good place to see the sunrise to help me figure out timing, logistics and help drop a calendar invite so I remember when I have to leave.

Yes, calendar use.

Opening Chrome. Going to Google. Searching.

AI overview by Google.

Great. So how far away is the location from my place? It's opening maps.

It's crazy. It's like a person, bro.

For the distance between my area and the hiking location.

Wow. It's Just like your own personal freaking agent, dude. It can just do it for you.

Searching for the sunrise time tomorrow.

I'm just blown away.

And is now dropping.

Crazy. So it just figured it out. It just did it all by itself. Figured it out. And like we've talked about, you know, calendar AIs and stuff like that before on the channel, but I, like I said, I've never seen anything this advanced in terms of just an all in one autonomous agent that can just use a computer. It's. It's mind blowing details, look, it puts in the specific details. It's not that fast, I'll tell you that. But obviously stuff like that's going to be improved over time, man. The implications of this are insane. Great.

It looks like Claude did it. This is a simple example, but we're sharing computer use early to learn from what people build.

Yeah, it's crazy, dude. Wild. So yeah, that's computer use in the demos that they showed off. Like I said, I have access today we're going to be messing with it. It's pretty limited right now. They have it in like an open public beta, but it's, it's well controlled. So I'm warning you there, this is going to be massive. And this is definitely leaps forward above OpenAI for example. OpenAI has never suggested that they have any sort of computer use AI yet. I'm just saying man, this, this puts anthropic on top in some ways. Sure they don't have the 01 self recursive, you know, intelligent models, but computer use, man, that's real world applicable stuff right now that is absolutely a game changer. Take a look at these benchmarks for the new Claude 3.5 Sonnet update and the Haiku model. Claude 3.5 Sonnet outperforms GPT4 Omni and Gemini pretty much across the board here on the math problem solving. Gemini 1.5 Pro seems to take it home. But this is four shot and then pretty much everything, it just sort of wins. And you'll also see that they have some new benchmarks to test the Agentic tool use, agentic coding. And obviously you know that these are the only models that actually are able to use these things, the Claude models. So they're the only ones that are benchmarked here. But it's nice to see that the sonnet models, Claude 3.5 sonnet old and new, steadily outperform the 3.5 haiku here. There's definitely a lot of room for improvement and I am excited to finally see Agentic Tool use as a new benchmark here for large language models or multimodal large language models, I could say, because it still needs to take that image input in to be able to actually use your computer. And is computer use a modality in its own. I guess so, kind of. You'll also see for Claude 3.5 haiku, this is a fast model, it is a small model, it's a cheap model, but it sort of kind of falls in between Gemini 1.5 Flash and GPT4O mini. Again, I guess the really big part here is that Haiku, the smaller model, well, it might not be all that much better than GPT4OH mini, and it might be a little bit worse than some benchmarks than Gemini 1.5 flash. It still has agentic coding, it still has agentic tool use, and none of these other models can do that yet. If you want to dive a little bit deeper into the weeds, obviously all the blog posts and stuff are going to be linked down below. They also have a whole nother blog post on just developing computer use and I like to see them give all of this information. Although at first glance here I'm not seeing a research paper, I'd like to see a more in depth research paper because I think that's better for the whole AI space at large. But again, this will also be linked down below. I think you guys are going to learn a lot more about computer use with this new Claude 3.5 sonnet model by us actually demoing it here today. Alright, so before we actually start messing around with this new computer use tool use model, let's talk about the other autonomous AI agents that have appeared before it and what their capabilities are and how that compares to this new one. As you guys can no doubt see, I have done quite a few videos on AI agents and out in the wild. These are going to be your two most notable ones. First we have the Hyperrite AI agent, which is pretty darn good. This mostly circles around just web use, but it can do things like order you a pizza, sift through ebay listings. And I think in this example they have it using LinkedIn. The coding stuff for example, is something entirely new though that requires you actually being able to use a computer in its direct form, not just using a website. And the same goes for Multi on AI, which is another agent that's very similar, you know. Here are the examples they provide here. How much did Nvidia stock gain today? Find the latest tweet from this person. Again, it's just sort of Only using like a web browser. So not necessarily fully using a computer. But again, they can do stuff like actually go through and pick out a bunch of computer parts and build you a fully working computer, not actually build it for you, but pick out all the parts so you could buy them, do research for you, etc, etc. And the thing is, with multi on AI and the hyperrite AI agent, I don't think that they can take in image inputs because for example, when I tested both of these, I tried to have them play an online chess game and it wasn't able to see the pieces or anything else. It was reading the website data some way that wasn't just taking screenshots, for example, like we see here with the Anthropic demo. And again, they can't actually use your computer, they can only use web browsers. That's sort of the big difference here. Now again, I'm not going to go through a whole tutorial on how to set up computer use yourself. It's really not that complicated though. So if you want a video on it, let me know. They've got a GitHub here. And this is more or less the demo itself. There's a couple of different ways to run it. You can do it with the Anthropic API Bedrock Vertex. And here is the computer use demo, folks. As you can see, it looks a little bit weird. Obviously this is running in a web browser, but this is like a local web page that's basically set up and running on my computer through something called Docker. And then it's connecting to the CLAUDE API through that website that's essentially running on my computer. And this is a virtual computer that it's running inside, so it can't damage your computer. Like I said, it's kind of restricted a little bit. They're keeping it within its certain bounds. Very limited at the moment, but still accessible. So let's type to Claude down here. Open Firefox. Can it do that? So now it's running the agent over there and you can see it's decently quick, but definitely not the fastest thing in the world. It goes down there and it sends a screenshot, then finds the Firefox icon and opens it up. You'll also notice that I can't control the screen here. I have to click this toggle to be able to control it myself. But we'll just give it like a brand new Firefox window. That's fine. And then I'll toggle the control off again. Okay, first thing is, first alright, first things first. Go to PC Part Picker and build me a Balls to the Wall AMD powered PC Enter. This is something like I said those other agents were pretty much able to do successfully every single time. So it's gon go over to PC Part Picker. It knows the website link right away. Let's see. It is now trying to use the website to the best of its ability. Okay, it's going and picking out a cpu. It's gonna pick out, you know, obviously some sort of AMD cpu, but we need to have a very powerful one because I said Balls to the wall, baby. So it's picking out the 7800x3D. I think that's a pretty good pick. That's a pretty powerful cpu, I'd say. I think it's having a little bit of difficulty right now figuring out how to get back to the rest of the Part picker. It's kind of stuck in this little window here. Okay. It's managed to navigate back to PC Part Picker and go back to the build. Okay, so I'm gonna stop you right there. There is some limitations that I'm seeing and it's because of the screenshots. If I toggle screen control on here, I'm going to move out and give this thing a little bit more to see here on the website. So it shouldn't struggle as much. There. There we go. We'll turn that back off so you can see a little bit more. Keep building me the PC. There is a limitation that comes with screenshots. If it could use your own computer, like if it could use my computer as it's set up right now, I think it would work a lot better right off the bat. Alright, it got stuck again. Build me a PC AMD very fast. There we go. Okay, Come on. Alright, we got an Rog Strix motherboard. That's a good one. I don't know why we did motherboard first, but you know, that's fine. I think a lot of the delays that we're experiencing here are just screenshot delays being uploaded to the Anthropic API and it has stopped itself once again. Alright, let's reset this thing. Let's try a better prompt. I'm giving it some extra information about how the website works. Maybe it'll work this time. All right, opened up Firefox. It's gonna start my build. Choose me a cpu. As you were. Right now it's trying to left click on something. I think it's trying to get back. Oh, now it's on the privacy policy. It's not great at using websites yet. Oh, and then we got a bunch of API errors. Number of request tokens has exceeded your per minute rate limit. Thanks, Anthropic. That's great. Well, that's aggravating, to be honest. So I don't know how it to the privacy policy. All it needed to do here was scroll down and then choose another part. CPU cooler. It's all that needed to happen. All right, continue building me a PC. Don't forget to use the scroll wheel to navigate. Okay, page down. You didn't need to do that. You didn't have to scroll down to find the motherboard section, man. Alright, I'll be real. I think they dumbed this demo down a little bit too much. I'd love to be able to just install this directly on my PC and just let it use my PC. I'll take the risk, man. I trust Claude. Yeah, I can't. Can't figure out how to go up. Yes. Oh, it's scrolling. Oh, and then it. We ran out of tokens. Too many tokens exceeded your permanent rate limit request. That's so aggravating. Okay, clearly this requires too many actions. It picked out the CPU though. So guys, if you're gonna build something like this and you're gonna have us be able to demo it, you need to allow us the opportunity to use it on our own PCs and to not run out of tokens immediately. How am I supposed to test this out when I get like immediately blocked by the restrictions you set in place? Alright, let's try something else. I'm taking this thing to Chess.com we're gonna play online. Play chess. I actually think it's gonna be decent at this. I am just continuously getting an error message, I think for token limits. Like I keep typing commands in and it's just like, no, not doing anything. I mean, you can see here in the API we're at the tier one rate limit. Larger organizations aren't gonna be restricted to that. But yeah, 50 requests per minute. Tokens per minute is 40,000. So it's going through quite a few tokens. Tokens per day. A million. Did we blast through a million tokens? I guess I wouldn't be surprised since it's just uploading images. Oh my. It's already used 600,000 tokens. Oh geez, what's that gonna cost me? Well, I mean, ideally it would click the play button and it would move the chess pieces. This demo is extremely limiting and it's pretty aggravating. We've also got some settings in this demo, like, you know, your API key, what model you want to use. You could use Haiku if you want. I figured I'd use the best model. I don't think that this is an issue with their actual framework they've developed. It's just the way they decided to demo it and release it publicly to us. Kind of aggravating. Like, yeah, sure, it can, you know, use a computer, but it's just this little virtual environment here. I'll open up like a paint canvas. Do you think you can draw me a smiley face? Draw me a smiley face. Oh, it's opening up Firefox again. I think it's trying to run the old stuff, the PC part picker stuff we told it to do. All right, it's going to Google. This is so aggravating. This demo is useless. Yes, computer use is a crucial step for AI models. The demos that they proposed on their YouTube channel they showed off look really promising. But it's clearly a different demo than what we've been provided right now because it's like early access and they're trying to keep everything safe. Just don't give us a demo then. Honestly, I'd rather wait. I'd rather wait for, like, a desktop app. That's what I really think I want. I get it. A virtual environment is safer. There's safety precautions. But this thing is useless in here, especially with the rate limits that you set on your own API. I also blasted through how many dollars worth of tokens. Oh, okay. Apparently that was only a few bucks in tokens at least. AI is becoming too cheap to meter folks anyways. Don't give us a demo we can't use. That's super aggravating. It looks really cool in the videos you showed off, and I believe that it does have those capabilities. I believe that it is the best computer AI interface that exists right now. But I can't use it. Your demo doesn't work. There's also a huge list of things that it currently refuses to do because of safety. This is Benjamin's post here on X. It is not allowed to create accounts on social media or other platforms. I don't get that. Send emails or messages we already have. AI is doing that right now on other platforms. It's not, you know, using a computer. Sure, but really post comments on social media. I guess they're trying to prevent it from being a bot, but I still feel like there's plenty of bot comments on all platforms, including YouTube. And is this really the most efficient way to do bot comments? Probably not. Make purchases. It's your fault if you're giving this thing your wallet and it's making purchases, man. I mean, if you're telling it to buy stuff, isn't that like your prerogative? Access private information? I guess that kind of makes sense. Complete captchas makes sense too. That's probably like genuinely a legal issue. Generate, edit, or manipulate images. Why make phone calls? There are already other AIs that actually do that autonomously. Restricted content, that's pretty broad. Perform actions that require personal authentication. Yeah, Marco makes a great point here. In other words, everything I would like for an AI agent to be able to do for me. Anthropic. Let the beast loose, man. You can build technology that is able to do all of these things, but if you never give anyone access to any of it, it's basically pointless. Right? I assume we're gonna get access to this thing in a broader scheme at some point, but again, this just makes me think, all right, I'll wait for open source to catch up. Because they will. OpenAI will catch up. Google will catch up. They're all gonna catch up. You guys have made a mark and I think that's good. You're showing off the research you've done and the progress you've made and it's. It's cool. It's cool. Don't even tease me with a demo that doesn't work all that well though. And the restrictions are crazy. The implications of a computer controlled AI are insane though. All. There's so much work, so much menial, annoying work that we all have to do on computers that these things could just take care of. It would be huge, it would be massive, and it looks really promising. This is like the. The briefest glimpse through the curtain that I've ever seen to the most game changing technology we've seen come out of AI in so long. So thanks, Anthropic, but you're really teasing me today and I'm a little upset about it. Peace out, guys. I'll see you in the next video. Goodbye.