Claude 3.5 Sonnet Upgrade + New API to Control Your Computer – First Look

Anthropic just released upgrades to a couple of their models and a new feature which they're calling Computer Use through their API. In this video we're going to quickly look at the blog post and also the API documentations for this new feature. Now these are not completely new models, but the first one is an upgraded cloth 3.5 sonnet, although it seems to be a major upgrade, and the second is cloth 3.5 haiku which is going to be available later in this month. Now there is no Update on Cloud 3.5 Opus. That's the one I think everybody is waiting for. And they also released a new feature which they're calling Computer Use. So it's an agentic framework that lets you control your computer through their Cloud API. We're going to look at that later in the video. So in terms of how these new models look like, they provided some benchmarks and you can see from the benchmarks that they are comparing the new 3.5 models with the original 3.5 Sonnet, GPT4, O Mini, Gemini 1.5 Pro and Flash. The new 3.5 Sonnet is beating all the other models on all the benchmarks except on math, but even the Haiku model is really performant and I would put it somewhere closer to GPT4 or Mini. The major upgrade seems to be related to coding ability of 3.5 Sonnet, which was a state of the art model when it comes to coding anyways and I think it's on par with the O1 preview or Owen Mini from OpenAI but they are highlighting the agentic use as well. So this model seems to be state of the art when it comes to agentic use on the coding benchmark which is this SUI Bench Verified it is state of the art model. The previously it scored only 33.4% but now there's almost a 50% improvement. And even the Claude 3.5 haiku, which is a much smaller model compared to Sonnet, is able to beat the original 3.5 sonnet, which is a pretty substantial improvement. Now the blog post talks about potential use cases, so as I mentioned, it's really good at agentic coding and tool use cases. So if you're building any sorts of agents or looking for a model that is really good at coding, the upgraded 3.5 Sonnet definitely gives you an edge over all the existing models. Now in terms of pricing there are no changes. So 3.5 Sonnet offers these advancements at the same price and speed as its predecessor. So you're not going to see any price changes, but there is a huge performance boost in terms of the updates to 3.5 haiku. I think it's a really good model for daily use cases, especially if it doesn't involve a lot of reasoning. Now this seems to be Even better than Claude 3 opus, the original opus in some use cases. But specifically for programming, it does seems to outperform the original 3.5 sonnet and the GPT4O model. So I think it could be a really good candidate if you're looking for a really good coding model, but at an affordable price range. So specifically, if you have a multi agent system and you're creating sub agent tasks which are very specific tasks that are performed by a smaller agent, I think 3.5 haiku could be a really good option there. And then you can have an orchestrator which is an agent that will orchestrate these subagents and that can be operated by something like 3.5 sonnet. Now, the model itself is going to be released later this month, so hopefully in the next seven to eight days because you're already almost towards the end of the month. And interestingly enough that a Claude Oranthropic is making sure that these models are available through a number of different API providers. So they will be hosting it through their own API plus Amazon Bedrock and Google Vertex. Now, the biggest announcement that has everybody excited is this computer use model. It's not a completely new model, but it's a new API feature and it's probably the very first feature offered by any API provider that lets you control your computer. And it can do clicks or type text or fill out forms. So here in the demo you provide a request to computer use through the Cloud API. It can access your computer, it can see what is on your computer screen and then can take actions on behalf of the user. Now it's a very interesting use case, but if you are using this new feature through the API, I think you need to be very careful because it can take different actions and in some cases even on their research it shows that it can deviate. It's in a beta right now, so it's not a fully ironed out solution. And that's why I think it's very important if you are going to run this, make sure to run it in a sandbox in a completely isolated environment because this can perform some destructive operations on your system. So it's available today through the API and they say that it's a beta feature. So please be aware that the computer use poses Unique risks that are distinct from standard API features or chat interface. So they recommend to use a dedicated virtual environment or container with minimal privileges to prevent direct system attacks and then avoid giving the model access to sensitive data such as login information or prevent information theft. Right. So it's a very interesting use case, but I think in very early days there are a number of different startups who are building agents specifically for controlling computers. And Cloud has their own take on it. So in terms of the API, it's a standard API system. Now you pass the computer screen information as a tool and then provide the user message the way you would do it in a normal API call to an anthropic model. The normal workflow is going to look something like this. You provide Cloud with computer use tools and a user prompt. Then just like the other tool usage or function calling, Cloud will decide to use that tool if it needs to. It can extract inputs, evaluate the tool on your computer and return results. And Cloud will continue calling computer use tool until it's completed the task. So the workflow is very similar to any other agent that you can build, but in this case the only difference is that it has access to your actual computer. So in some cases you want to have a separate computer or you want to limit the different operations that the API can perform. One more thing which is a sort of attack that we have seen before is that CLAUDE will follow commands found on in content, even if the conf if it conflicts with the user instructions. So for example, CLAUDE instructions on pages containing images may override instructions or cause CLAUDE to make mistakes. So this is one of the earlier attack that we had seen with the multimodal models. People were embedding messages in the images that you would pass on to these multimodal models and they would follow those instructions. It poses a very interesting challenge specifically for agents that are going to be designed to use computers. Like what happens if there are any malicious instructions on a web page that these systems are going to be accessing. So there are interesting challenges that needs to be resolved before these systems can be extremely helpful. The other thing that I'm thinking about is we might have to design our interactive systems in a very different way moving forward. The way we are interacting with computers today probably is going to look very different in the future if you are building these multi agent systems. So in some cases we'll have to re architect our systems so that these LLM based agents or whatever form it looks like can interact on our behalf more efficiently. It doesn't have to be the way humans interact with these digital systems. This was a first quick look at the new announcements. Very interesting things. I probably create some videos on computer use, although I need to decide whether I want to run this on my own machine or have a sandbox. So stay tuned for that video. Thanks for watching and as always, see you in the next one.