The Next Step in Our Journey to AI Agents: Anthropic's Computer Use

Today on the AI Daily Brief, the next step towards our agentic future. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. To join the conversation, follow the Discord link in our show Notes. Hello friends, Quick note Before we dive in today, instead of our normal setup where we have headlines followed by the main episode, we are just doing a long extended main episode that is all about this big advance from anthropic product. I think you'll understand why as you dig in, but that is why you're not getting any headlines. Enjoy and later this week we'll be back to more normal formats. There have been two moments over the past few weeks where we really get a chance to step back and see the very beginnings of a branch off on the AI evolutionary tree. The first of those was of course, OpenAI's 01. This was their reasoning model. And it wasn't just a bigger model than GPT4. Oh, it takes a fundamentally different approach to how it works. O1's model basically has a built in sort of chain of thought approach that breaks down complex tasks into simpler steps and reasons through them sequentially before generating a response. This is why, by the way, one of the things that we Learned when prompting O1 as opposed to GPT4O is that you don't need to add things like think step by step. Now the net impact of O1's reasoning approach is that it's much better able to handle things like coding and math. And when it comes to business, it's better about things that have a distinct right answers based on inputs. So it might not make a poem any better than GPT4O, for example. But if you're trying to figure out the ideal arrangement of a banquet hall for a big convention and you can give it all of the relevant inputs, it's going to be much better at figuring that out than, for example, GPT4O. And while the difference is subtle, the important thing, like I said, is that it represents this branch off of the LLM tree where we are moving subtly but surely into a new reasoning era, which is of course in and of itself the beginning of a new agentic era. OpenAI also recently shared their stages of artificial intelligence. Level one was chatbots and AI, with conversational language. Level two, which O1 represents, were reasoners with human level problem solving. Level three, which we're not at yet, are agents or systems that can take actions. And then level four and five are basically what collections of agents can do and more advanced capabilities. So Level four is Innovators AI that can aid in invention and level five is organizations AI that can do the work of a full organization. This is the relevant setup for what Anthropic recently announced, which is called Computer Use now. This is part of a larger announcement that also included some model updates, including an upgraded Claude 3.5 Sonnet as well as a new model Claude 3.5 Haiku. But there's no doubt that the main discussion and excitement around this was computer use developers. Anthropic rights can now direct CLAUDE to use computers the way people do. By looking at a screen, moving a cursor, clicking and typing text, they write that Claude 3.5 Sonic can now follow a user's commands to move a cursor around their computer screen, click on relevant locations and input information via a virtual keyboard. Basically, it emulates how people interact with their computers. Now, the version of this that they started exploring is purposefully very general. It is not use case specific and that seems very intentional. They write. A vast amount of modern work happens via computers. Enabling AIs to interact directly with computer software in the same way people do will unlock a huge range of applications that simply aren't possible for the current generation of AI assistants. Now, in a similar way to the idea that O1 was not just a bigger or better model, but a different approach, so too is computer use. Not innovation as model innovation, but a capabilities innovation. Anthropic writes. Over the last few years, many important milestones have been reached in the development of powerful AI for example, the ability to perform complex logical reasoning and the ability to see and understand images. The next frontier, they argue, is computer use. AI models that don't have to interact via bespoke tools, but that instead are empowered to use essentially any piece of software as instructed. Their announcement post also gave a little bit of the background around how this came together. One of the cool behind the scenes things is how this works. They write When a developer tasks CLAUDE with using a piece of computer software and gives it the necessary access, CLAUDE looks at screenshots of what's visible to the user, then counts how many pixels vertically or horizontally it needs to move a cursor in order to click on the correct place, they continue. Training Claude to count pixels accurately was critical. Without this skill, the model finds it difficult to give mouse commands. So this is a big part of the secret of these capabilities is that it literally counts pixels. However, they also found that these new capabilities unlocked a lot of new capabilities that it wasn't specifically trained on. They say we were surprised by how rapidly Claude generalized from the computer use training we gave it on just a few pieces of simple software such as a calculator and a text editor. In combination with Claude's other skills, this training granted it the remarkable ability to turn a user's written prompt into a sequence of logical steps and then take actions on the computer. We observed that the model would even self correct and retry tasks when it encountered obstacles. Basically in the same way that the ability to use a computer changes how we think, the ability to use a computer seems to change the way that the LLM quote unquote thinks. Anthropic sums up this shift by saying computer use is a completely different approach to AI development. Up until now, LLM developers have made tools fit the model, producing custom environments where AI's use specifically designed tools to complete various tasks. Now we can make the model fit the tools Claude can fit into the computer environments we all use every day. Our goal is for Claude to take pre existing pieces of computer software and simply use them as a person would. Now, a couple caveats to all of this. First, Anthropic is quite clear that this is very experimental at this stage and that it tends to be pretty error prone. It also says that there are a bunch of actions that seem incredibly easy or effortless for people that are difficult to impossible for Claude's computer use. Those include scrolling, dragging and zooming. The framework in which they're releasing this is as an experiment and right now it's only available through API. This isn't something that a general user of Claude can just fire up and do. It's something that's going to require a developer to actually set up a specific application for. Overall, the tone for Manthropic is very much this is a first glimpse into the future, not a production ready product. They even joke. Even while recording these demos, we encountered some amusing moments. In one, Claude accidentally stopped a long running screen recording, causing all footage to be lost. Later, Claude took a break from our coding demo and began to peruse photos of Yellowstone national park. So we can say for the machines that at least it has good taste. Now I should note that Anthropic also does get into some of the questions around safety. They did not, for example, train computer use on navigating the Internet. Yet they note that Claude 3.5 Sonnet, even with computer use, still remains at AI safety level 2, meaning that it doesn't require a higher standard of safety and security measures than those we currently have in place. They continue when future models require AI safety level 3 or 4 safeguards because they present catastrophic risks. Computer use might exacerbate those risks. We judge that it's likely better to introduce computer use now, while models still only need AI safety level 2 safeguards. That means we can begin grappling with any safety issues before the stakes are too high, rather than adding computer use capabilities for the first time into a model with much more serious risks. For what it's worth, this sounds very similar to how OpenAI always talks about their approach to safety. In other words, iterative deployment that allows all of us to adapt to new capabilities in something of a more incremental way. Today's episode is brought to you by Venice. Venice is a private, uncensored, generative AI app. It accesses open source models to enable text, image and code generation without the fear of being spied on or having your data exploited. Discuss anything with Venice without concern about it being monitored, sold, or given to advertisers and governments. Venice is different because your conversations and creations are kept securely within the browser, never stored or accessible by Venice. Unlike other AI apps, Venice won't tell you what's okay to say or not. Venice won't patronize you. It simply provides direct access to machine intelligence. No topics are off limits, no ideas are taboo. With Venice, you're in control of the AI as you should be. Pro subscriptions are available for $49 a year or $8 per month. AI Daily Brief listeners receive a 20% discount on Venice Pro. Visit Venice AI NLW and enter the discount code NLWDAILYBRIEF. That's NLW Daily Brief. All one word, two things I feel qualified to talk about organization and productivity, apps and AI tools, which is why I am very happy that today's episode is sponsored by Notion. Notion combines your notes, docs and projects into one space that's simple and beautifully designed. And the new Notion AI has the capability of multiple AI tools built in, which means you can search, generate, analyze and chat all inside Notion. The new Notion AI is a single AI tool that does it all search across Notion and other apps, generate docs in your own style, analyze PDFs and images, and chat with you about anything. Notion is a perfect place to organize your tasks, track your habits, write beautiful docs, and collaborate with your team. The more content you add to Notion, the more Notion AI can personalize its responses for you. Basically, unlike generic chatbots, Notion AI already has the context of your work. There are also a bunch of great integrations. Notion uses AI knowledge from GPT4 and Claude and with AI connectors, which is now in beta, Notion AI can also search across Slack discussions, Google Docs, sheets and slides and more. Tools like GitHub and Jira are coming soon. Notion is used by over half of Fortune 500 companies, but more importantly, it's used all day, every day by me. Try Notion for free when you go to notion.com aidaily brief. That's all lowercase letters notion.com aidaily brief to try the powerful, easy to use Notion AI today. And when you use that link, you're supporting the show once again. That's notion.com aidaily brief. Today's episode is brought to you by Super Intelligent Every single business workflow and function is being remade and reimagined with artificial intelligence. There is a huge challenge, however, of going from the potential of AI to actually capturing that value. And that gap is what superintelligent is dedicated to filling. Superintelligent accelerates AI adoption and engagement to help teams actually use AI to increase productivity and drive business value. An interactive AI use case registry gives your company full visibility into how people are using artificial intelligence right now. Pair that with capabilities building content in the form of tutorials, learning paths and a use case library, and superintelligent helps people inside your company show how they're getting value out of AI while providing resources for people to put that inspiration into action. The next three teams that sign up with 100 or more seats are going to get free embedded consulting. That's a process by which our Super Intelligent team sits with your organization, figures out the specific use cases that matter most to you and and helps actually ensure support for adoption of those use cases to drive real value. Go to BeSuper AI to learn more about this AI enablement network. And now back to the show. Alex Albert, who is, I think, nominally the head of Developer Relations, but lists himself as the head of Claude Relations on Twitter, wrote a nice thread about how big a shift this represents. He writes, computer use is the first step towards a completely new form of human computer interaction. In just a few years, the way we interface with computers will be completely different from today. Computer use allows AIs to use computers just as you would. No complex abstractions or specific APIs, just pure visual understanding and interaction. Exactly like how you use your computer. He gives an example video where he says, Claude opens up Claude AI in a browser prompts it, opens the outputted website code in a new code file within Vs Code, and then proceeds to fix a bug in the website. All with computer use Alex continues, this is entirely different from how most agentic frameworks currently work today. Most quote unquote agents are a patchwork of multiple bespoke APIs glued together under the hood of some complex scaffolding. Alex says, I believe we'll be able to reach near human level performance in the next few years, if not much sooner. When that is reached, it means AIs can operate the basics of a computer just as well as an average person can. At that point we can start stringing together AIs doing tasks. Now, instead of an AI doing a simple task on a computer that would only take a human a couple of minutes, it will do that task, then move on to two more tasks. Suddenly the AIs will be doing end to end tasks that would take humans hours and days. Read a 50 page research report to create a full executive summary and slide deck. Scan financial documents to build a DCF model. View a wireframe to ship a production ready website. Alex continues, Combine this with a longer context window and increased chain of thought and now you have the beginnings of the unhobbling of AI products. The pieces of the true agent puzzle are starting to fall into place. If you're developing on AI today, you need to be thinking about building complementary pieces to this reality because it may come faster than most expect. AI entrepreneur Bindu Reddy writes, Computer Use API by Anthropic is an interesting take on agentic APIs. Agents are challenging because they have to talk to other systems and most of these systems don't have good APIs. One potential solution is to use the Computer Use API, which allows the LLM to pretend to be a human operating a computer. The biggest issue with this approach is security, but it's not insurmountable. Now. What about the use cases that are available right now? Obviously there's a lot of talk about things in the future, but what can computer use do at this moment? Well, once again, Alex Albert writes fun story from our time working on computer use. We held an engineering bug bash to make sure we found all the potential problems with the API. This meant bringing a handful of engineers in a room together for a few hours. We were hungry, so one of our engineers first computer use requests was to ask Claude to navigate to DoorDash and order enough food to feed a group of people. About a minute later we saw Claude decide to order us some pizzas. Alex Volkov, an AI evangelist with weights and biases and host of the Thursday AI podcast, writes, mind officially blown. Once again, this computer used Claude Demo from Anthropic didn't work for me, so I just asked it to fix itself. So it did. Kaylin KS writes Anthropic recently released Computer Use API with which developers can direct Claude to use computers the way people do. He then shares a video around how he set up computer use to use the firecrawl dev API to gather information and fill out job applications. Michel Catasta, the president at Replit, writes, I can't tell you the last time I was so excited to see a new AI capability in action. We plugged in Claude Computer use in Replit Agent as a human feedback replacement and it just works. I feel like it won't take long until our agent will become fully autonomous. Now Replit had some advanced access to this, so has had a little bit more of a chance to experiment. To the extent that there was any skepticism, it was not about the long term, but about the very short term. Joe Mueller, for example, responded to the announcement asking, is this just a faster horse? Developer Tony Gee writes, first look at Anthropic's Claude computer use demo. Feel like it's cool, but kind of underwhelming and definitely not cost effective yet. 150,000 tokens just to visit and navigate through a couple sites. Either the models have to get much cheaper or your use case has to be insanely valuable. Or am I just missing something? Others are concerned about just how much change we're going to go through. Elena Alfaro writes, the new computer mode by Anthropic is the beginning of an era where we humans will stop understanding software. The interfaces we have today will become useless. Now I think it's a valid concern, but also an open question. How much and in what specific ways does it matter for us to understand interfaces? And like I said, there is a ton of experimenting already happening. Stetson Blake writes, if this is the slowest and cheapest that agentic automation will ever be, we're in for a wild future. I tried a few different tasks. One involving administering my Facebook group's posts and declining and accepting member requests, which it did great at. I tried booking a haircut. It seemed to work, but hit API rate limit, did the make yourself a website thing from the demo and it cooked. Ultimately, Chubby on X summed up a lot of the sentiment when they wrote, this is so huge. Today we entered the era of agents. First Microsoft, now much more so. Anthropic today marks a new step in the direction of AGI Remove OpenAI Anthropic is in the lead. Mackay Morin writes, the dawn of truly agentic AI is upon us. While today we're seeing the early stages of APIs enabling models to interact with systems, we are rapidly approaching an era where intelligent and autonomous agents will be deeply integrated into our operating systems, serving as intelligent assistants that can truly understand and execute our intentions. I have long maintained that the future of computing isn't just about smarter algorithms, more parameters, or improved tuning. Rather, it's about intelligently automated systems that can actively engage with our digital environment. The recent advances in multimodal models and computer interaction capabilities are just the beginning of a new trend. As these models become more integrated into our operating systems, we will see a fundamental shift in how we interact with technology. This is both society altering and inevitable. And so that is Anthropic's computer use. Still nascent? Yes. Still limited in what it can do. But as with O1, another break off the branch of the evolutionary tree of LLMs and AI, and another step towards a very different agentic future. That's going to do it for today's AI. Daily Brief Appreciate you listening as always and until next time. Peace. Sam.