Claude 3.5's New AI Agents Are GAME CHANGING (Claude 3.5 Agents + New Models)

So today Anthropic put the AI industry on hold when they announced their upgraded family of 3.5 models. You can see here it says, today we're announcing an upgraded Claude 3.5 Sonnet and a new model Claude 3.5 Haiku. The upgraded Claude 3.5 Sonnet delivers across the board improvements over its predecessor with particularly significant gains in coding, an area where it already led the field. And Claude 3.5 Haiku is matches the performance of Claude 3 Opus, our prior largest model, on many evaluations for the same cost and similar speed to previous generations of Haiku. Now there's a ton of announcements to get in, but the main things are going to be Claude 3.5 and of course their incredible new agentic computer use, it says. We're also introducing a groundbreaking new capability in public beta computer use available today on the API. Developers can direct Claude to use computers the way people do by looking directly at a screen, moving a cursor, clicking buttons and typing text. Claude 3.5 Sonnet is the first frontier AI model to offer computer use in public beta and at this stage it is still experimental and at times cumbersome and error prone. We're releasing computer use early for feedback from developers and expect the capability to improve rapidly over time. So a lot to get through, but that's just a brief overview on what they've announced released. So with the upgraded Claude 3.5 Sonnet, let's actually take a look at what Anthropic have released with Claude 3.5 Sonnet because this is the model that most people are going to be excited about considering. It is the model that is leading the way in terms of coding and many other areas. So when we take a look at the Claude 3.5 Sonnet benchmarks, we can see that once again Anthropic have somehow managed to surpass every other state of the art model in its similar class respective to GPT4O Gemini 1.5 Pro, we can see the Claude 3.5 is managing to excel in all domains. We can see in the graduate reasoning level, the GPQA diamond. This manages to surpass GPT4O and Google's Gemini 1.5 Pro and the MMLU Pro. We can also see that Claude 3.5 leads achieving a 3% gain over its previous benchmarks. In the coding eval, we can also see that Claude 3.5 Sonnet manages to do better again taking it further than the GPT4O benchmarks. And in the high school math competitions we can see that there is actually A rather large jump from the Claude 3.5 sonnet new to the Claude original 3.5 sonnet. Almost a double in performance in the high school math competitions. So it seems that Claude's mathematics has gotten a serious boost in the visual qa. The mmmu, which is the visual understanding, this one has bumped up around 2% to a level that is now state of the art. And one of the things that that Claude actually mentioned was that they wanted to improve the vision capabilities. So that way when you're feeding in CLAUDE screenshots so that it can use your computer, the model doesn't struggle. So this was one of the things that is really incredible. Now there is a fascinating piece of the benchmarks in which there are only areas for Claude specific models. I'm guessing that the other models haven't been tested on these capabilities because they might not have the capacity to even do these things. So let's take a look at what this says. It says on agentic coding on the swebench verified, Claude 3.5 sets a really incredible standard for what the state of the art is. We can see that this model gains a 49% benchmark compared to the prior version of Claude 3.5 sonnet of 33.4%. Now, I think this is rather incredible and I want to show you all a screenshot that is simply going to potentially blow your mind. We can see here that Alex Albert says that the most significant gains are in coding. The new 3.5 sonnet sets a new state of the art on the software engineering bench, verified with a score of 49% using no complex scaffolding, besting all models, including Reasoning models like OpenAI 01 Preview and specialized models for agentic coding. I think this kind of achievement cannot be underestimated considering the fact that this is a general purpose model that also excels at a variety of different things and now surprisingly manages to even beat a reasoning model at coding that I'm sure OpenAI didn't think this would even happen. Now, one of the craziest things about this leaderboard is is that the software engineering benchmark has constantly being broken almost every single month. We can see that every month or so there's around a 5% gain on this benchmark, whether it be from a new scaffolding, which is essentially how you structure the models in some agentic form with tool use, or with an entirely new base model that just has an internal system that is just so advanced that you can use it to achieve better responses and thus better results. We can see that Claude manages to best this, getting up to 49%. So this was one of the things that I truly think is absolutely incredible because it manages to even beat out many specialized models for coding. We can also see that for agentic tool use on the airline benchmark, we can see that Claude 3.5 Sonnet's new benchmark is up at 46%, a 10% gain on Claude 3.5 Sonnets, 36%. Now, one of the things that I do think about this, that we can take away quickly from Claude 3.5 Sonnets Benchmarks is that when we look at the new areas of agentic coding and agentic tool use, we can see that other models aren't even on the radar. What I mean by this with this agentic benchmark area is I think that this is going to be the standard for new AI models. Oftentimes we see benchmarks that are in the gpqa, the MMLU and math. But I think what's going to become more important as time goes on is how these models perform in agentic areas. As you all know, 2025 and the following years onwards will likely be the years for agentic AI. So those models that are showing early promise in agentic capabilities now are going to be the models most loved by the AI community. So Claude 3.5 sets a new level that these other companies are probably going to have to follow suit, which makes this an even more competitive race. Now, what was also fascinating was that we also did get a new model class, which is Claude 3.5 haiku. So 3.5 haiku replaces 3.0 haiku as the fastest and least expensive model. Surprisingly, it outperforms many state of the art models on coding tasks, including the original Claude 3.5 sonnet, which is insane to say, and of course GPT4O. So we have Claude 3.5 haiku. Although it is not available today, it will be made available in the coming weeks. But that model having a large agentic tool use, a 51.0 on the retail and 22.8% on the airline benchmarks, is absolutely incredible, considering the fact that this model is going to be the fastest model and of course the least expensive model. So we're now basically getting a clawed 3.5 sonnet level model that seems to be faster than the predecessor and of course a lot cheaper. So for those of you who are developers who are trying to build certain applications, but cost may have been a factor, Claude 3.5 haiku might just be the saving grace for what you are trying to build, considering the credits for this kind of model isn't going to be that expensive. Now, of course, next we must cover these agentic capabilities and how they actually work. Anthropic have released a few videos which I will showcase here so that you can understand exactly how well these models do when it comes to agentic use. So Claude have stated that with computer use, they're trying to build something fundamentally new. Instead of making specific tools to help Claude complete individual tasks, they're basically teaching it general computer skills, allowing it to use a wide range of standard tools and software programs designed for people. Developers can use this nascent capability to automate repetitive processes, build and test software, and conduct open ended tasks like research. And to make these general skills possible, they've built an API that allows Claude to perceive and interact with computer interfaces. Developers can integrate this API to enable Claude to translate instructions. For example, use data from my computer and online to fill out this form or computer commands. For example, check this spreadsheet, move the cursor to open a web browser and navigate to the relevant web pages, or fill out a form with the data, and so on and on. Osworld, which evaluates a model's ability to use computers like people do. Claude 3.5 Sonnet scored 14.9% in the screenshot only category, which is notably better than the best AI system score of 7.8. And when afforded more steps to complete that same task, Claude scored 22%. So in this first video, this is where Claude is being shown to be used for automating operations.

So I'm Sam and I'm one of the researchers here at Anthropic Computers is something that we felt was going to be important for a while now. So today we're going to be talking about a very early version we have of computer use and walking through a representative example of the things we think it's going to be useful for. We're going to be going through a quick demo today. In this fictional demo, a customer, in this case the Ant Equipment company, has come to us and asked us to fill out a vendor request form. The data I need to fill out this form is scattered in various places on my computer. What we're going to do is ask Claud to look at the spreadsheet, check if Ant Equipment is in there, and if not, move over to the CRM and try and find some more information there. Once it has this data, Claude's going to then fill out the form for us and hopefully transfer the information across to the vendor form. The first thing that's going to happen is Claude's going to start taking screenshots of my screen and quickly realizes that the Ant Equipment company isn't actually in the spreadsheet. So the first thing it does is it swaps over to a CRM and searches for the company we're interested in. Luckily, we get a search match and Claud then starts scrolling through the page looking for all the information it needs to fill out this form. Claw then autonomously starts transferring the information across without me having to do anything, and goes through the steps and fills out all the information needed. And then submits the form. This example is representative of a lot of drudge work that people have to do. This is available in the API. We're excited for people to try it, and we should expect things to get a lot better over the coming months.

Next, we have an example of Claude 3.5 sonnet being used to control the computer and execute a few coding tasks to a website.

I'm Alex, I lead developer relations at Anthropic, and today I'm going to be showing you a coding task with computer use. So we're going to be showing Claude doing a website coding task by actually controlling my laptop. But before we start coding, we need an actual website for Claude to make changes to. So let's ask Claude to navigate to Claude AI within my Chrome browser and ask Claude within Claude AI to create a fun 90s themed personal homepage for itself. Claude opens Chrome, searches for Claude AI and then types in a prompt asking the other Claude to create a personal homepage for itself. Claude AI returns some code and that gets nicely rendered in an artifact on the right hand side. That looks great, but I want to make a few changes to the website locally on my own computer. Let's ask Claude to download the file and then open it up in VS code. Claude clicks the Save to File button, opens up VS code, and then finds the file within my Downloads folder and opens it up. Perfect. Now that the file is up and running, let's ask Claude to start up a server so that we can actually view the file within our browser. Claude opens up the VS code terminal and tries to start a server, but it hits an error. We don't actually have Python installed on our machine, but that's all right because Claude realizes this by looking at the terminal output and then tries again with Python 3, which we do have installed on our machine. That works. So now the server's up and running. Now that we have the local server started, we can go manually. Take a look at the website within the browser and it looks pretty good, but I noticed that there's actually an error in the terminal output and we also have this missing file icon at the top here. Let's ask Claude to identify this error and then fix it. Within the file, Claude visually reads the terminal output and then opens up the Find and replace tool in Vs code to find the line that's throwing the actual error. In this case, we just ask Claude to get rid of the error entirely. So it will just delete the whole line. Then Claude will save the file and automatically rerun the website. So now that the error is gone, let's go take a final look at our website and we can see that the file icon has disappeared and the error is gone as well. Perfect. So that's coding with computer use and Claude. This took a few prompts now, but we can imagine in the future that Claude will be able to do tasks like this.

Another example of Claude being used to complete tasks on a computer that the average person might find extraordinarily helpful.

I'm Pooja and I'm a researcher at Anthropic. I'm going to show you a simple example of computer use today. My friend's coming to San Francisco next week and I want to take him to do some touristy stuff. I think doing a sunrise hike with a view of the Golden Gate Bridge never gets old. So I'll ask Claude to figure out some logistics for us. I'll ask Claude to find a good place to see the sunrise to help me figure out timing logistics and help drop a calendar invite. So I remember when I have to leave, it's opening Chrome, going to Google searching. And it looks like it's found something. Great. So how far away is the location from my place? It's opening maps, Searching for the distance between my area and the hiking location. Cool. So now it looks like Claude is searching for the sunrise time tomorrow. And is now dropping it into my calendar. And populating it with some details. And great. It looks like Claude did it. This is a simple example, but we're sharing computer use early to learn from what people build.

Now, one of the most interesting things that I personally saw from this release was a short video in which Anthropic state that even while recording these demos we encountered some amusing moments. In one, Claude accidentally stopped a long running screen recording, causing all footage to be lost, which seems like the AI managed to make a mistake there. And it says later Claw took a break from our coding demo and began to peruse photos of Yellow National Park. I think this might be one of the emergent behaviors. I do wonder if AI is going to do some unexpected things on your computer. So it seems like you will have to make sure that when you set out your tasks that you don't just leave this thing entirely running. Although I'm sure Anthropic will leave all of the documentation that you need to ensure safety on the API page. For example, if you do see the API page, it does state that computer use is a beta feature. Please be aware that computer use poses unique risks that are distinct from standard API features or chat interfaces. These risks are heightened when using computer use to interact with the Internet, and to minimize these risks, consider taking precautions such as using a dedicated virtual machine or container with minimal privileges to prevent direct system attacks or accidents. Avoid giving the model access to sensitive data such as login information to prevent information theft, limit Internet access to an allow list of domains to reduce exposure to malicious content, and ask a human to confirm decisions that may result in meaningful real world consequences, such as tasks requiring affirmative consent, such as accepting cookies, executing financial transactions, or agreeing to terms of services. In some instances, CLAUDE will follow commands found in content even if it conflicts with the user's instructions. For example, CLAUDE instructions on web pages or contained in images may override instructions or cause CLAUDE to make mistakes. We suggest taking precautions to isolate CLAUDE from sensitive data and actions to avoid risks related to prompt injection. Finally, please inform end users of relevant risks and obtain their consent prior to enabling computer use in your own products. So overall, it seems that this is something that is right really in beta, meaning that you do need to ensure you are completely careful with this model. There are still mistakes it can make. Prompt injection can happen, and CLAUDE in some scenarios may just ignore your wishes completely. So it's important to understand exactly what is going on.