First Reactions: Claude 3.7 Sonnet and Claude Code

Today on the AI Daily Brief Anthropic has just launched Quad 3.7 Sonnet. Before that in the headlines, ChatGPT has hit 400 million weekly active users. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. To join the conversation, follow the Discord link in our show. Notes. Welcome back to the AI Daily Brief Headlines edition. All the daily AI news you need in around five minutes. Quick note, for the next couple of episodes we will be audio only and this week we'll be back to our normal video format as well. We kick it off today with an announcement from OpenAI at the end of last week that ChatGPT has hit 400 million weekly active users, surging a full 33% since December. OpenAI hasn't previously disclosed these figures, which show the service is still growing at a rapid rate. Chief Operating Officer Brad Lightcap posted, ChatGPT recently crossed 400 million weekly active users. We feel very fortunate to serve 5% of the world every week. 2 million plus business users now use ChatGPT at work and reasoning model API use is up 5x since the O3 mini launch in January. That last number, I think is hugely significant. O3 mini has kicked up API reasoning model use 5. X. Lightcap added that GPT 4.5 and 5 are coming soon, with plans to offer unlimited use of GPT5 to free users on low inference settings. In comments to cnbc, Lightcap discussed the gulf between hundreds of millions of free users and relatively slow business adoption, stating there's a buying cycle there and a learning process that goes into scaling an enterprise business. AI is going to be like cloud services. It's going to be something where you can't run a business that ultimately is not really running on these powerful models. Underneath the surface, however, the implication, which is completely true from our experience at Superintelligent is that it just takes time. Even the most obvious things in the world come up against human and organizational inertia that has to be pushed through. Turning to other topics, Lightcap discussed the deep SEQ moment as validation that AI has entered the zeitgeist rather than as a negative for OpenAI. He commented, Deepseek is a testament to how much AI has entered the public consciousness in the mainstream. It would have been unfathomable two years ago. It's a moment that shows how powerful these models are and how much people really care. Many people pointed out when they saw these numbers that if this sort of rate of growth increases apace, we are going to see a billion ChatGPT users in extremely short order. Speaking of GPT 4.5, some companies are getting ready. The Verge is reporting that GPT 4.5 could be released as soon as this week, and according to sources familiar with Microsoft's plans, the company is already readying server capacity for GPT 4.5 and GPT 5. They expect GPT 4.5 to be released imminently. GPT 5, on the other hand, is expected to launch in late May, aligning with Microsoft's Build Developer conference. This could represent a much closer working relationship between Microsoft and OpenAI for this year's releases. Microsoft was reportedly blindsided by the release of GPT4O last May. It offered voice and translation services as well as a big speed boost, all at a cheaper price than Microsoft services built on GPT4 Turbo. It took Microsoft until October to overhaul their services to catch up with OpenAI, who is of course supposed to be their biggest partner here. Now, there have been obviously lots and lots of rumors about the potential breakup of Microsoft and OpenAI, but it appears that in this case at least, Microsoft has been given the heads up this time around, and so presumably we could expect copilot updates ready to go shortly after OpenAI's releases. Sam Altman, meanwhile, has been hyping it up, posting last week that trying GPT 4.5 has been much more of a feel the AGI moment among high taste testers than I expected. GPT5, meanwhile, will remember, be a much larger rethink of the company's product line. It'll be the first model to integrate reasoning and non reasoning into a single model. OpenAI have also suggested that they will devise a way to apply the right amount of inference to each query, doing away with the need for the model selector. Already the rumors are starting to build. Lisan al gaib suggested that OpenAI could already be testing GPT 4.5 in public, routing some O3 mini queries to the new model. Meanwhile, OpenAI rumor monger Riley Coyote passed on whispers that Wednesday will be the release day. Now, speaking of new models, there is a little bit of controversy swirling around Grok3's benchmarks, with some doubting the new model from XAI is really a match for OpenAI's O3 mini. The controversy deals specifically with the AIME benchmark, a set of competitive math problems. XAI tested their model using a method known as cons 64 or best of 64. This involves generating 64 responses and selecting the one that appeared most frequently. Best of 64 is a well accepted benchmark standard, so there's no issue with using it per se. The problem was that XAI compared their result against O3 mini's benchmark using a one shot solution method referred to as pass at 1. OpenAI had presented this one shot benchmark to demonstrate that O3 mini was better than 01 even when the older model made 64 attempts. In other words, XAI wasn't making an apples to apples comparison. It appeared particularly galling to the OpenAI team as Xai was promoting Grok3 as the world's smartest AI. Boris Power, the head of applied research at OpenAI, posted, Disappointing to see the incentives for the GROK team to cheat and deceive in evals. TL Dr. 03 mini is better in every eval compared to Grok3. Grok3 is genuinely a decent model, but no need to oversell. Tony Wu, the co founder of XAI, commented, Obsession with metric pass at 1 is just stupid. To compare fairly you have to fix the test compute budget and without disclosing what test time compute method is used behind O3 mini, we cannot really compare. At the end of the day it's just about which one is a better product. Also, depending on the product, for example Consumer product versus API, you may have different requirements in terms of latency or total flops. For test time compute. Try Grok 3 and tell me if you think it's better or worse than O3 mini. Now this discussion, which on first glance one could be forgiven for viewing as just the inherent competitiveness of two teams, did spill over into the rest of the AI research community who discussed how to deal with benchmarks moving forward. Tierratax has compiled all of the available benchmarks in a single chart with both OneShot and best of 64 variants commenting, I actually believe GROK looks good there and OpenAI's test time compute chicanery behind 03 mini high pass at one deserves more scrutiny. Matthyn Lambert wrote, I think it's safe to say that Xai and OpenAI both have committed minor chart crimes with thinking models. Frankly, there are no industry norms to lean on. Just expect noise, it's fine. May the best models win. Do your own evals anyway. AIME is practically useless to 99% of people and this I think is for sure the key point. Every model still pummels us over the head with these benchmarks as soon as they release their newest thing saying look, we've improved, blah blah blah blah and it fundamentally doesn't matter. I'm sorry but at this point I am fully on the train that these benchmarks are totally soaked. There is almost no relevant signal in that that all of the models now are at the very high end of these things and that they just tell you almost nothing. I hope we get some more good work on thinking about new types of evaluation because we desperately need it. But at this stage I think that there's no other reasonable answer if you're willing to take the time and the resources to do it, than to just try every type of query and every type of prompt and every type of challenge against all of the state of the art and see which one does best that. Or alternatively just pick one, assume that it's going to be close to as good as the state of the art and will be as good as the state of the art in a couple of weeks when they ship the latest update. Speaking of which, I think that leads perfectly to our main episode topic, which is Anthropic's launch of Claude 3.7 Sonnet. Today's episode is brought to you by Vanta Trust isn't just earned, it's demanded. Whether you're a startup founder navigating your first audit or a seasoned security professional scaling your GRC program, proving your commitment to security has never been more critical or more complex. That's where Vanta comes in. Businesses use Vanta to establish trust by automating compliance needs across over 35 frameworks like SoC2 and ISO 27001, centralized security workflows, complete questionnaires up to 5x faster, and proactively manage vendor risk. Vanta can help you start or scale up your security program by connecting you with auditors and experts to conduct your audit and set up your security program quickly. Plus, with automation and AI throughout the platform, Vanta gives you time back so you can focus on building your company. Join over 9,000 global companies like Atlassian, Quora and Factory who use Vanta to manage risk, improve security in real time. For a limited time, this audience gets $1,000 off vanta@vanta.com NLW that's V A N T A.com NLW for $1,000 off. If there is one thing that's clear about AI in 2025, it's that the agents are coming. Vertical agents by industry Horizontal agent platforms Agents per function if you are running a large enterprise, you will be experimenting with agents next year. And given how new this is, all of us are going to be back in pilot mode. That's why Superintelligent is offering a new product for the beginning of this year, it's an Agent Readiness and Opportunity audit. Over the course of a couple quick weeks, we dig in with your team to understand what type of agents make sense for you, to test what type of infrastructure support you need to be ready, and to ultimately come away with a set of actionable recommendations that get you prepared to figure out how agents can transform your business. If you are interested in the Agent Readiness and Opportunity audit, reach out directly to me. Nlwsuper AI Put the word agent in the subject line so I know what you're talking about and let's have you be a leader in the most dynamic part of the AI market. Welcome back to the AI Daily Brief. Anthropic has just launched Quad 3.7 Sonnet, what they call their most intelligent model to date. Similar to how OpenAI appears to be describing what GPT5 is supposed to be, Anthropic calls this a hybrid reasoning model that quote, produces near instant responses or extended step by step thinking. One model, two ways to think. Now holding aside whether it actually does that well, it is extremely telling. I think that this is the new norm going forward. No more the separation between reasoning and non reasoning models. It's just one model to rule them all that can navigate between the two. Now of course, as you would expect, Anthropic announced a bunch of benchmarks to demonstrate how Claude 3.7 Sonnet is a big improvement over its predecessor. They showed an increase in performance on everything from GPQA diamond, the Graduate Level Reasoning to the aime. I've just been on my rant about evaluation benchmarks so I won't repeat that again. Ultimately I think what you can say is that even based on their own sharing, in most of these cases it is a nudge forward rather than a leap forward. The one exception to that which we'll come back to is around coding where the Sweep Bench verified test saw a huge improvement from 49% with Claude 3.5 sonnet all the way up to 62.3 to 70.3% with Claude 3.7 sonnet. Agentic tool use was also way up, showing a meaningful increase in performance over Claude 3.5 sonnet as well as OpenAI's 01. Indeed, this is what led Anthropic to say that Claude 3.7 is state of the art model for both coding and agentic tool use. They write in developing it we've optimized somewhat less for math and computer science competition problems and instead shifted focus towards real world tasks that better reflect the needs of our users. So at least someone is hearing these rants about benchmarks and what we should be thinking about now. It's very clear that coding is the whole ball game right now for Anthropic, so we're going to come back to that in a moment. But before that, let's get some first reactions. Rowan Chung from the Rundown writes, anthropic just dropped Claude 3.7 Sonnet, the best coding AI model in the world. I was an early tester and it blew my mind. It created this Minecraft clone in one prompt and made it instantly playable in artifacts. Professor Ethan Malik writes, it is very, very good. Its Vibe coding from language is impressive. Here's a one shot prompted video game based on the Melville story Bartleby the Scrivener Box's Aaron Levy writes, Box has been doing evals on it with enterprise docs and it's extremely strong at hard math, logic, content generation and complex Reasoning use cases BoxAI will support Claude 3.7 Sonnet later today in the Box AI studio, Adonis Singh writes, dude, what? I just asked how many Rs it has. Claude Sonnet 3.7 spun up an interactive learning platform for me to learn it myself. And indeed, while the general impressions were favorable, it's because a lot of those impressions were about coding. Cjzz writes, Claude Sonnet 3.7 is built for coders. Don't evaluate it on web search and multimodality evals. Claude is doubling down on what they know the best AI coding. Matt Schumer shared the SW Bench verified benchmarks and said, this seems to be a huge step up flower slop. Claude 3.7 seems to be way ahead in coding compared to 0103 mini high, R1 and Grok3 according to my first Vibe test. A test I like is whether a model can build a fully functional Doodle jump clone from scratch. It's right at the edge of what soda models almost get right, but not quite until now. 01 tried, but the window closed instantly with a console error. 03 mini high made a basic version, but platforms were too far apart to reach. R1 had no starting platform, so you'd just fail instantly. Grok3, even with extra thinking, also crashed instantly. Claude 3.7 nailed it first try one prompt fully working with the prettiest design and even a funny little doodler. It simply just did it without any flaws or bugs. And indeed, this is perhaps why that was not the only part of the announcement. Head of Claude Relations Alex Albert writes, we're opening limited access to a research preview of a new Agentic coding tool we're building. Claude Code. You'll get Claude powered code assistance, file operations and tasks execution directly from your terminal. After installing Claude Code, simply run the Claude command from any directory to get started, ask questions about your code base, let Claude edit files and fix errors, or even have it run bash commands and create git commits. Alex continues Within Anthropic, Claude Code is quickly becoming another tool we can't do without. Engineers and researchers across the company use it for everything from major code refactors to squashing commits to generally handling the toil of coding. He shared a message from Slack that said, I just want to say Claude Code is very quickly taking over my life and becoming my go to tool. Truly think there's something very special here. Pietrono explains it a little bit further. Claude Code is a command line tool that lets developers delegate substantial engineering tasks to Claude directly from their terminal. In early testing, Claude completed tasks in one pass that would normally take 45 minutes of manual work. Not Adam Paul writes Claude Code is an in terminal coding agent and it's objectively the coolest thing a Frontier company has shipped since GPT4. Here I get it to read my project specs and tell me what's left to implement against the code base. Haven't even started coding with it yet and I'm hooked now to the extent that anyone had any concern it was around Price. Harrison Kinsley writes, claude Code is really nice. The UI is so wonderful. I like the action type rules. Well done. Prepare to spend up to $5 an hour running it. Potentially more deja vu. Coda responded more like 5 USD per 20 minutes. Others, like Anthropic's Catherine Olson, jumped in to talk about where it wasn't perfect. She writes, Claude code is very useful, but it can still get confused. A few quick tips from my experience coding with it@anthropic 1. Work with a clean commit so it's easy to reset all the changes. 2. Sometimes I work on two dev boxes at the same time, one for me, one for Claude Code. We're both trying ideas in parallel and so on and so forth, and I actually think that this is a super valuable category of information. Not only does sharing this stuff build trust with your users, it also guides them to use your tools more effectively. Overall, I tend to agree with Benjamin Decracker who writes, I have a hunch that Claude Code, the terminal coder, is a bigger deal than many people realize. Certainly there is a sense that combined with the other updates. We are in the middle of another big shift. Professor Ethan Malik again just published a new piece on his One Useful Thing blog called A New Generation of Claude 3.7 and Grok 3. Yes, AI suddenly got better again. For tomorrow's episode, I'm going to be doing a look at what's evolving faster and what's evolving slower in AI than people might have imagined, and so we'll definitely be coming back to some of this assessment. For now, though, I'm excited to go dive into Claude 3.7 sonnet myself, and I hope that when you test it out, you come back and tell us what you found as well. For now, that is going to do it for today's episode of the AI Daily Brief. Appreciate you listening as always. And until next time, peace.