New Claude AI Takes Control of Your Computer!

Wow. So Anthropic hit us with an announcement today that nobody really expected. They released a brand new version of Claude 3.5 Sonnet, the best model when it comes to code generation. And they even introduced a new API that allows Claude to control your computer. Meaning the LLM can go ahead and click buttons, scroll around, use the browser, do things for you, all based on text prompts. What details matter here? What is the availability of this and how can you get your hands on this brand new computer Use API, which is a first in AI space. People try to build things like this, but Anthropic is the first major company to actually release this. Let's get into it, starting with the model updates because those are quickly covered. So they announced two big updates here. Claude 3.5 Sonnet is getting a new version with improved benchmarks across the board. Sonnet was already impressive, but they managed to increase the scores across the board, especially when it comes to coding and math related issues. With the release of 01 preview, some people were debating if that model is better for certain coding use cases. I was always of the opinion that Sonnet 3.5 was still state of the art and your go to model if you wanted to generate code. But with this update they just solidified their position as these benchmarks improved 5 to even 50% in some cases. And just keep in mind these are numbers that were already beating some of the other models like 4.0 before. Now the gap is even larger. This is not the type of move that OpenAI would do. They only release things if they really have to. CLAUDE is just pulling even further ahead here with this new Sonnet 3.5 release. The good news is that you can actually use this right now in the Claude web interface. As you can see if I go into here, I have the Claude 3.5 Sonnet new preselected. Now I'm not sure why they didn't name this Claude 3.6 or something like that, but there you go. Claude 3.5 sonnet new available right here. And again, even yesterday I thought that this was the best LLM to use for code generation. Now it's even better. And the news behind the news here is that all of these agentic frameworks like Replit Agent or Pythagora that we covered on the channel here are going to perform even better as all they do in the background is talk to models like this. So that's a Sonnet update. And by the way, the API is equally as fast and costs the same as it did before the second announcement, here is about their small model Haiku being updated to the 3.5 level. And this jump is just massive compared to the numbers before. I mean, heck, some of these numbers come close to GPT4.0 performance. Look, on code generation, GPT4 has 90.2, Claude 3.5, Haiku is 88. And the speed and the prices of this aren't matched. I mean, this is their GPT4.0 mini competitor. Now, the thing to be noted here is that, that this is not available yet. They said this will be released over the coming weeks, but honestly, as this is the least significant part of this announcement for most people, that's totally fine. And now let's talk about the big news here, which is what they're calling computer use. So here's the base concept. You go in and you give it a prompt. Hey, go ahead and check out when the sunrise happens tomorrow in my location. And then create a calendar event in my calendar for a sunrise hike. And it figures out all the steps it takes, create screenshots along the way, it clicks the buttons that it needs to click, it Googles what it needs to google. It takes the data, it moves it around, it can switch between applications and it just gets it done. Now, obviously this is the very first version of this that we have seen out of one of the AI companies. Now, I'm not saying this was not possible before. People built these versions that combine multiple capabilities. But this is the first one that is officially coming out of an AI lab and it comes with some demo videos which I definitely recommend you watch. I'm not going to go through all of them here. Here you can just check him out on the blog post that is linked in the description below. But the first one is essentially showing how it takes data from one spreadsheet and then heads on over to the browser and enters it into a CRM, something that a human would need to do manually. This thing can just automate with a series of prompts. Now here's the thing. Whenever it gets a specific task done, then he reprompts it and gets to the next task. This is really visible in the second demo here, which you can see in the Twitter threads that I will also link below. And this one is more development focused, akin to something like Replit Agent that we talked about here before, before. And it goes ahead and based on one prompt, it creates a website. But then it also does the critical steps of actually taking that code and copying them over to VS code, running the code in there, hosting the site locally so you can access it through the browser. And then when he checks out the website, he actually figures out that there's some mistake on it and he re prompts the computer user API to fix the mistake inside of the code editor. Not just the code, it updates the whole app. And then the last example is similar to something I outlined earlier where it checks out maps data, it goes to Google, it finds the information on the sunrise and it creates a new event inside of the calendar. Now this is all pretty amazing, but they do note that hey, this is really just step one. It will go wrong a lot of the times. It will be very limited. But the amazing thing about all of this is we have access to it now and what the implications exactly are and where we will see this in action, we don't know yet. Time will show that. And once we do, I will create one of those AI use case videos you guys seem to love on this channel. Something like 20 use cases for Anthropics computer use. But I usually like to take my time with those to really give people people to build interesting things. So once that happens we will review all the use cases. But as I mentioned, you can use this thing today. Now how do you use it? Well, here's a few ways, let me show you. So this is through API only, so you will not find this inside of the CLAUDE interface. But they did publish a GitHub repo which is Anthropic computer use demo. Now this takes a few steps and will not be feasible for non technical people. You have to run it inside of a Docker container and then you can try it out yourselves. But I did a little bit of digging and it actually worked to together with REPLIT before the launch of this thing as REPLIT will be integrating this into the product. And replit together with the launch of this actually has a space where you can try this thing right now. Now you do need the Replit subscription which I believe is $25 a month, less if you go yearly. But if you have that you can head on over to this link that I will also link below and they have Anthropic computer use demo. And this thing couldn't be any easier to use. All you need to do is go down to the bottom left here. When you're logged in, head on over to Secrets and instead of Secrets you just copy paste your API key from Anthropics console that you can get in here. And once you did that you can actually click run in here and this will start up and you can start Using it. Now, you do have to start here in the web view and give it some sort of commands. So let's just give it something that would be relevant to me right now as I'm in Brazil for a kite surfing workation type of situation. And I'll just tell it something that is top of mind, like find the wind forecast for the next seven days in Guru and then find a car rental site in this region. I don't know, I'll just run this and let's have a look at what happens here. We need to head on over to this output pane to see it actually work. And here when I expand it, I believe we should see a browser and all of this should be happening autonomously. Let's see if it worked. Okay, it seems to be working, but it wasn't showing me the output. So let me try this one more time. Input the same prompt, say run. Okay, the agent is running and it's already using the tools. It's taking screenshots, as we mentioned. It's working with those screenshots. And here in the output pane, we should actually see it work live, which is pretty cool. Okay, so it's inside of Firefox. Now it should find a website with the wind forecast. Okay, Google searching. Windy.com guru. That's good. That's actually the app that I use. It's got the spot name right too. I mean, I gave it to it, but fair enough. It clicks that. And if we check the agent view, this is good. This is progressing very well. You will see that it's actually taking screenshots here. It's running prompts, it's taking actions, it's using the computer to mouse move to specific coordinates and then it uses the left click to actually go in there. You see, all of this is happening here in the background fully autonomously. And it's already on here. So this is very interesting. It's using the search again. And as you can see, this works. And people will need to experiment with these, People will need to build with this. But it's a very exciting time because this is really the moment of inception of autonomous agents. We've been moving in this direction more and more and OpenAI has been teasing us with things like O1 preview that can reason and advanced voice mode, that is a really good voice interface. But they never gave us this. They didn't give us the ability, the power to actually remote control a computer with LLM. And now we have it from Anthropic. It's the first version and in the next days and weeks, we'll show how useful this thing actually is in practice. But think about it. Learning more about this is a literal superpower. If you know how to prompt a little bit and you know how to use this thing, you can just set up multiple agents and have your own army of plot 3.5 sonnet instances run around and do things with computers on the Internet. You can have them do tasks, you can have them do research. You could even have them run a service business if you come up with the right workflow and the right prompts to go along with that. I'm super excited about this. Can't wait to see what else this brings. And that is pretty much the update. And if you want to learn more about this, I'll actually be running a community event on this in about 10 days every month I run LLM Innovations and this is the perfect topic. So we'll dive deep on this and teach our community members how to set this up for themselves. Because I personally really believe in this agentic direction. I think although these things might be primitive right now, if you look close enough you will find the use cases that matter and that can make a difference in your life. And me and the team are here to support you on your way. Okay, that's all I got for today. Leave a comment below with what you will be using this for and I'll see you soon.