EP 534: Claude 4 - Your Guide to Opus 4, Sonnet 4 & New Features

This is the Everyday AI Show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business, and everyday life. Right at the end of the busiest week in AI ever, Anthropic decided to drop two big new AI models on us all. As if we weren't busy enough with everything else that we had just seen released from Microsoft, Google, and others, we now had two new contenders in Claude 4 Opus and Claude 4 Sonnet to play with and to see how good these models are and if they can actually grow our companies and our careers. So just like we did with everything Microsoft and everything Google, we're going to be breaking down what's new, with this new Anthropic Claude 4 release and talk about, is this going to be your new large language model you use every day? Or is this maybe just for software engineers? Or is this not just a good model? All right. So we're going to be going over that today and a lot more on Everyday AI. What's going on, y'all? My name is Jordan Wilson. I'm the host of Everyday AI. And if you're looking to grow your company and career with generative AI, then this is for you. This is your daily livestream podcast and free daily newsletter helping us all learn and leverage generative AI. So if you haven't already, please go to youreverydayai.com. So there you're going to get the recap of today's show in our free daily newsletter, but also there at youreverydayai.com, you can go listen to, watch, and read more than 530 back episodes sorted by category. So no matter what you're trying to learn, whether it's sales, marketing, HR, ethics, data analysis, whatever it is, we've got probably dozens of shows in all of those categories talking to the world's leading experts. It is a free generative AI university. So make sure you go check that out. All right. Most days we go over the AI news. I didn't want to make this a super long show. So that's going to be in today's newsletter. So make sure you go sign, sign up and grab all of that. All right. Livestream audience. It's good to see y'all. Like Marie says, good morning, AI family. Yeah. If you're listening on the podcast, we do this live almost every single Monday through Friday at 7:30 AM Central Standard Time. I'm in Chicago, so you can do the math there or maybe have Claude do the math for what time that is. But, you know, join, come, come hang out, you know, with, People like Josh Cavalier saying good morning from Charlotte, North Carolina. Jordy joining us from Jamaica. Love to see it, Jordy. Jose from Santiago, Chile. We got some international flavor. I love this. Brian joining us from Minnesota. Everyone else, Big Bogey on the YouTube machine. Christopher joining us from Bowling Green, Kentucky. Thanks for joining. But let me know as we go along, What are your thoughts on the new release on Claude 4? But right now we're going to— this is your guide. This is the basics. We're going to start here. All right, so like I said, Claude had their first ever— or sorry, Anthropic had their first ever developers conference this past Thursday. And there they announced, among other things, their two new flagship models in Claude 4 Opus and Claude 4 Sonnet. And I'm already getting a little bit confused saying those things out loud. So I did talk about this a little bit on the show yesterday, but they even changed their naming mechanism. Whereas before it was, you know, Claude 3.7 Sonnet, uh, was the last Sonnet variation, but now it's just, uh, Claude Sonnet 4. So, you know, now the number is at the end. So a lot of new things, even how they're changing or naming their models. But, you know, if you are brand new and if you don't know too much about Anthropic's Claude, it is and has historically been usually a top 3, you know, AI lab along with, you know, OpenAI, Google. Microsoft is kind of in a different category, but it's one of the biggest large language models in the world. Although most people, unless you're a real AI kind of dork or a heavy large language model user, you might not know Cloud. And I think that is actually, whether you're saying fortunately or unfortunately, only going to become intensified. Just, I think fewer and fewer people are actually going to be using and hearing about Cloud, I think, because I think they're getting away from being a general chatbot company. But more on that here in a couple of minutes. But, you know, there's 3 big, there's 3 variations of Claude. So you have your biggest model, which is Opus, your medium model, which is Sonnet, and then you have your small model, which is Haiku. And you'll notice that only the Opus and Sonnet models got updated to the 4 variations. So Claude Haiku 3.5, which is their smallest and most efficient model, did not get updated. So that is still Claude Haiku 3.5. So I guess the only thing that got updated was the naming mechanism there. So here's a quick overview of what is actually new. All right. So we have hybrid reasoning. So this is an instant and extended thinking mode for flexible reasoning. So, you know, we talk about kind of two types of large language models here on the show. Yes, I'm overgeneralizing this, but you have your traditional transformer, your old school large language models, which is funny to say something's old school. But those are ones that just kind of snap something back to you real quick. And then you have these models that are reasoners, or they can think step by step. They can show logic like a human and plan ahead. So these models, you know, Gemini 2.5 Pro is a reasoning model. The OpenAI O models, O3, O4, O1, those are all reasoning models. So Claude 4, it's a hybrid model. So it just, sides on how much it should think and should it just spit things out to you really quick. It is a top coding model. That is by far where Anthropic is seemingly focusing on and kind of abandoning general use. But it is now state of the art in coding. It will be interesting to see how long they hold that state of the art coding title. I don't think it's going to be long if I'm being honest, because Google could come in with an update literally any second now. And probably wipe a good majority of these benchmarks that Anthropic is now hanging their Claude hat on. So another big thing is tool integration. So using external tools like web search during the reasoning process. So that's if you are, you know, there's two different ways you can look at this, right? So using it on the front end as a front end user, right? So if you go to Claude AI or Claude.AI, right? So using it as an AI chatbot and then obviously if you're building on top of it, or using a service that uses Claude's API. So there's always a front-end user, which is your more non-technical people, and then the back-end people that are maybe building on top of Claude's API. But regardless, you can have this new tool use during the reasoning process, which is big, right? And this is nice because it catches Anthropic up with OpenAI and Google in that regard. Also now there's long-running tasks. So I haven't personally seen this, and I think this is only if you're using it in the API, but Anthropic is saying that it can, the new Claude 4 models can maintain coherence on complex tasks for extended periods. They talked about Claude running a task, I think Claude 4 for like 7 hours on the API side, which is absolutely bonkers. Now that you have, you know, models literally like punching in the clock and they're like, yeah, I'm going to go work a 7-hour day now. I would never give a model that complex on the backend because yes, it's going to require obviously the API and Claude 4 is one of the most expensive APIs, at least when we're looking at general use case large language models. And I would never want something like that to happen where it goes out and it works on a long task for a long time and then, okay, what happens if it times out, right? Did you just waste, I don't know, couple hundred dollars, you know, having Claude go code for 6 or 7 hours straight. I'm not sure. And if you do want to get a taste of Claude, and if you're not on their paid plan, they do offer very, very limited, very limited options for Claude 4 on the free plan. All right. But let's be honest, I'm going to call a spade a spade. All right. So I think, you know, the paid plan is like, you know, $20 a month for the pro plan. And even on that, you can barely use the thing, right? It started as a joke, but now it's just sad for Anthropic as a company. I routinely will hit this rate limit. I'm on a paid plan. I paid $20 a month for Claude Pro, and I will routinely hit the rate limit in about 4 to 10 minutes almost every single time I try to use, even preparing for this show, hit it within about 7 minutes. So it's laughable. So yeah, I even chuckle more that there's a free version of Sonnet. So I don't know. I venture to think if you look at the free version the wrong way, you've hit your rate limit. So if you think that this is anything like a model that you can use, like, you know, Google's Gemini, you know, ChatGPT, Copilot, anything else where you have generous limits and it can be your partner in whatever type of work you're doing, absolutely not. If you're on a base $20 a month plan, a Teams plan, the limits are a little better, but the free plan, yeah, I, it's probably just a marketing gimmick. I don't even know if it could, you know, take a long prompt, with a lot of context. It would probably not work if I'm being honest. Right. All right. Let's keep this thing going. And by keeping this thing going, should we do another show? I want to give everyone a fair shake. And yes, I'm not the biggest Anthropic Claude fan. I broke down why. About 6 months ago. I'll have to pull up that episode number. But hey, livestream audience, if you do want a second show, because I've been doing, you know, multiple shows when, you know, Google comes out with a new model, when OpenAI comes out with a new model. So if you do, let me know right now. Just tell me what, show A, show B, show C, show D, or show E. Okay. And I'm going to throw this up again at the end. So show A, why Claude is losing the AI chatbot race. Show B, real-world use cases for Claude 4. Show C, Claude 4's improved artifacts, how to use them. Show D, don't do any more Claude. Jordan, stop. No more Claude. Or Show E, you can just pitch a Claude show in the comments. So livestream audience, if you could help us out, or podcast peeps, you can always subscribe to the newsletter or in the show notes, I always have our email. My LinkedIn, and you can, you can let me know what you, what show you want to do. So let me know, but I'll throw this up again at the end. So maybe after we go through everything that we have right now, you can let me know which show is, is that one. Oh, I did do a pretty— I'll say a teardown maybe of Claude and why your company should not be using it in episode 400. So if you want to go listen into that. That's Anthropic Claude: Why Your Business Shouldn't Use It. And I would say a lot of those reasons still hold true to today. So yeah, if you want one of those shows on the screen, go ahead and shout it out. All right. So let's talk about the benchmarks. This is what Claude is— and Anthropic, sorry— is really hanging its hat on is specifically software engineering, right? If you haven't noticed, they've kind of abandoned the everyday business professional, right? Which is kind of sad because a year or so ago, I think the Claude models were among the best in the world for everyday business leaders. Today, not really. I don't think unless you're a developer, unless you're in software engineering, or unless you have an edge use case. Right. I know a lot of people love Claude for like writing content. Right. But if I'm being honest, if you do a little bit of prompt engineering, OpenAI's GPT-4.5, better, and the limits are better. And then Gemini 2.5 Pro, better, limits are better. Right. I think Claude got this. It was crowned very early on. Right. Because at the time, you know, the other large language models were really bad at writing in general. Right. Everything was just ultra robotic. Still, you know, a lot of models are by default, and Claude still is pretty good. You know, if you're trying to zero-shot you know, some decent copywriting. But hey, as someone that got, that's been getting paid to write for 20 years as a former journalist with a little bit of prompt engineering, Claude is not better. Uh, OpenAI's model and, uh, Gemini's model are better. And the benchmarks say that, right? But people that maybe are a little bit lazier, right? And they don't wanna like do any work and they just wanna just go in and spend like 4 seconds, uh, inside Claude and be like, write something amazing. You're right. Claude will usually give you a better first draft if you don't do any work on the front end. But if you do any work on the front end or if you iterate with it a little bit, yeah, Claude's not that good. All right. But what it is really good at is software engineering. My goodness. So for our podcast audience, I have a screenshot here from the Claude 4 release looking at SWE-Bench Verified. So this is a benchmark for performance on real-world software engineering tasks and Opus 4 and Sonnet 4 are both scoring in the 72 percentile here on SuiBench, whereas the previous Sonnet model, the best one, 3.7, scored a 62%. So a pretty big jump here, but not that far ahead of other models, at least with baseline. You know, we're talking a 72%. They have parallel test time compute scores, which I'm not going to count those. That's essentially like, you know, trying over and over trying to squeeze the most juice, right? But if you're comparing apples and apples, yes, Opus 4 and Sonnet 4 are the best models for software engineering, but it's not by a whole lot, right? We're talking 72.5 for Opus 4, and actually Sonnet 4, the quote-unquote medium model, did slightly better at 72.7. But OpenAI is right behind there with their Codex-1. That's their new kind of coding-specific model with a 72. OpenAI's O3 with a 69, and then you have Gemini 2.5 Pro with a 63. So it's not like their lead is insurmountable, but by default, it is the best large language model in the world for software engineering. And I think that is where Anthropic is really focusing. But when it comes to just general usage, general intelligence, so, you know, sometimes we talk about the LM arena, which you put in one prompt, and you get 2 outputs. You don't know which model they are. You vote for the best one that gives you an ELO score. So right now, Claude 4 doesn't have enough info yet to be on the LM Arena, but I don't expect it to be anywhere near the top. But when looking at good third-party benchmarks that pull in multiple evaluations, such as Artificial Analysis Intelligence Index, that's what I have on my screen now for our livestream audience, right? So this is a good third-party, I would say, pretty much unbiased. This is pulling in 7 different benchmarks, right? So MMLU Pro, GPQA Diamond, Humanities Last Exam, Live Codebench, SciCode, AIM, and Math 500. So it's pulling in these different scores from widely used benchmarks in the LLM space. And right now, Claude 4 Sonnet, even with thinking mode enabled, is coming in at, what's that, number 8? Yeah. So, you know, like everyone that says, oh, Claude 4, best model in the world. It's like, for what? Right. So unless you're in software engineering, unless you're a developer, a coder, right? Yeah. Then it's the best model. But I wouldn't expect that to be for long because I would expect, you know, probably both Google and OpenAI to come in within a couple of weeks and swoop that away from Anthropic. And with Anthropic's recent, right, the last year and a half of their update cycle, They're not updating as quickly. They're not shipping as quickly as OpenAI and Google. So especially if your business, especially like on the backend for the API, if you're trying to make a long-term decision, the API, it's very pricey. We're going to get to that here in a minute. And also for all other use cases, as we see here with artificial analysis index, it's not very close. Claude 4, Sonnet, Thinking, it's not really there. Right? It's not really there. It's not a top model. So, I mean, we'll see these obviously change as models get updated, but, you know, on this artificial analysis intelligence index, the top models are number 1 is O4 Mini High from OpenAI, then Gemini 2.5 Pro from Google, then O3 from OpenAI. So, you know, yeah, no one's— that's why I like when people are like, oh, Claude's the best general use case model. I'm like, No, right? I don't know why people want to argue with, with science and math and stats. I don't know, maybe it's fun to do on Twitter or something. All right, let's get into all the details, y'all. So here's kind of the, the, the launch, right? So here's, here's what we got. So like I said, this was announced last week, Opus 4 and Sonnet 4 models open for, sorry, Opus 4 is the flagship for more complex tasks and coding excellence, even though, like we said, Sonnet is benchmarking pretty much like at the same. So there's not a big difference, at least right now, in Sonnet 4 and Opus 4, whereas primarily there was usually a pretty big gap between this medium and larger model. So Sonnet 4 offers more balanced performance for general and high-volume use, and both employ that hybrid reasoning for instant responses or deep reasoning. AI moves too fast to follow, but you're expected to keep up. Otherwise, your career or company might lag behind while AI-native competitors leap ahead. But you don't have 10 hours a day to understand it all. That's what I do for you. But after 700+ episodes of Everyday AI, the most common questions I get is, Where do I start? That's why we created the Start Here series, an ongoing podcast series of more than a dozen episodes you can listen to in order. It covers the AI basics for beginners and sharpens the skills of AI champions pushing their companies forward. In the ongoing series, we explain complex trends in simple language that you can turn into action. There's 3 ways to jump in. Number 1, go scroll back to the first one in episode 691. Number 2, tap the link in your show notes at any time for the Start Here series, or you can just go to starthereseries.com, which also gives you free access to our inner circle community where you can connect with other business leaders doing the same. The Start Here series will slow down the pace of AI so you can get ahead. All right. Let's talk about some of the new features, advanced tools, reasoning, and memory. So extended thinking with tool use is huge. So that includes web search and code execution. You also have now parallel tool execution, which is very important now for a baseline large language model to have that allows it to use multiple tools simultaneously and swap between those while it's reasoning. So now Anthropic is on board with that. Memory files are created to maintain context over long duration tasks. So that is something I'm interested to test a little bit more. For me, I'm not usually a fan of these, you know, memory type files with the large language models. Same thing with ChatGPTs. I have it disabled. One of the main reasons is I use large language models for everything. right? I use it for myself, my multiple businesses, multiple clients, multiple things in my personal life, right? So the whole memory is not always good because sometimes I might want Claude to out— or, you know, a large language model to output something, you know, super long and informal. And sometimes I might want something, you know, very, very short and choppy, right? Sometimes I want something that's, you know, visually rich. Sometimes I want literally strict bullet points and it varies, you know, so if you are only using large language models for one very specific purpose, you might find some utility with this new Claude 4 kind of memory file for me, or if you are a power user using large language models for everything, maybe not so much. There's also now the thinking summary that shows condensed reasoning, but you can see the full chain of thought in developer mode. Kind of in Claude's sandbox. All right. It is, and it's crazy now we're saying only, right? So we're talking about context window. It is only that 200,000K token context window. So Opus can output 32,000 tokens at once. Sonic can output 64,000 tokens at once. So that's essentially how much Claude 4 can remember at any given time before it starts to forget things. So this is a little bit better than OpenAI's ChatGPT, but it is far behind Google Gemini when you look at those 1 million token plus context windows. So the brain or being able to remember something, not as impressive, even though Claude was an original leader in this longer context space. I think a lot of people were hoping or looking for a couple of things with the new Claude 4. They were hoping for a longer token context window, which we didn't get, and they were hoping for reduced API prices, which we also didn't get. All right. There's also the new API includes code execution and MCP connector for external systems. That was huge for our developer and more technical friends. Right. But for everyday business users, especially if you're using Claude on the front end, nothing, nothing to see there. The files API does simplify document handling for repeated referencing across sessions and extended prompt caching up to 1 hour improves agent workflow efficiency. So yes, if you are building on top of these models on the backend, building agentic systems, you know, trying to swap models in and out. Yes, I will say that Claude 4 is very capable in that regard as well, not just from software engineering, but when you're looking at a model to power agentic workflows, you have to look at Claude 4 as well. And you see the prices and then you go look at Google and OpenAI's prices and then you're like, yeah, wait, why am I looking at this? It doesn't make sense. Like we talked about, some of the SWE benches, Opus and Sonnet are really just state of the art there. For other models are showing 65% less shortcut taking in agentic tasks versus Sonnet 3.7. And I think that's a big one, right? I follow the agentic space very, very closely. and a lot of people with SONNET 3.7, which was just released a couple of months ago, were pretty disappointed with its ability, uh, to follow longer, uh, tasks. So it did show that these Claude 4 are taking way fewer shortcuts in agentic tasks, which I think is huge. Uh, and then you do have those high compute options, which does boost scores across the board. All right. The other thing, Claude Code. All right, so now almost all companies are coming out with dedicated, you know, like a dedicated IDE, you know, dedicated coding tools, something that you can use, you know, on your desktop. So Claude Code is for developers. So this is a little separate than if you're using Claude.AI on the front end or building on top of Claude on the back end. This is a dedicated piece of software for developers to code and work with their code base. So Claude Code is now generally available with VS Code and JetBrains plugins as well. And it is now the preferred model for GitHub Copilot. It has the extensible SDK and the very popular MCP connector. So yeah, Anthropic's Model Context Protocol, it is wildly popular, right? If, if, if, which is kind of crazy to say, like if I look at everything Anthropic over the past year, probably the biggest news or the most promising advancements out of Anthropic It's not these coding models. It's not, you know, Opus-4, Sonnet-4. It's not Claude Code. It's not any of these things. It's probably the MCP connector. So this allows different, you know, agentic systems and different large language models to talk to each other on the internet. So it's a language, how websites have API, you know, APIs. AI systems and large language models, agentic AI, couldn't talk to each other, right? So it was really Claude that blazed the path. And now the other big players, including Google, Microsoft, and OpenAI, do support the MCP connector. So that's huge. And then also Cloud Code, like we talked about, it does enable that autonomous multi-file code refactoring over extended period. Yeah. So their example was it can work for literally up to 7 hours autonomously. You know, if you do have a super large codebase inside Claude Code. Yeah. It's just, I don't know. I want someone to make like a funny, you know, VO3, you know, short on Claude Code literally showing up for a 9 to 5 and everyone's like, you know, hey, AI is nothing like working a 9 to 5. And then, you know, you have Claude Code punching the clock and, you know, taking a lunch break and everything like that. All right. Here's the other disappointing thing in the thing. If you are looking on the API side, you gotta look at the costs because it looks like everyone in the large language model space is having this race to almost like ridiculously free compute, right? Compute too cheap, or, you know, intelligence too cheap to meter. Everyone in the world except for Anthropic. Their costs are absolutely bonkers. So Opus 4 is priced at $15 per million tokens input and $75 per million tokens output. So yeah, yikes. SawNet 4 costs $3 per million input and $15 per million output. So for comparison, I'll bring up the pricing for— let's see, I had it up here. I'll have to pull it up here. The pricing for, I mean, Gemini and OpenAI, it's significantly, significantly cheaper, right? And this is where a lot of people were disappointed and were hoping, you know, a couple of updates, you know, everyone wanted out of Claude 4, they wanted a longer context window, number one, they wanted more features, more capabilities, which I think we got that. And number three, they wanted cheaper pricing for people using it on the API side. And we didn't get that. So I'm gonna look up here just for comparison, the price per token for Google Gemini 2.5, and also we'll do GPT-4o because yeah, it's $15 and $75. If you're— it's just not sustainable anymore. right? Uh, if Anthropic had an insurmountable lead in any of these categories, that it made sense for companies. And, and so why, like, why do you care? Like, why should you care about this, right? If you're just logging into Claude.ai, you don't need to care about this, right? You're paying your $20 a month. You're, you know, the rate limits are absolutely terrible. The product is great, right? The rate limits are terrible. So a lot of people are, you know, companies specifically when they're wanting to build on top of Claude, their API and, you know, people in the software development space. So maybe they're using Cursor or they're using, you know, these tools and then bringing their API key and building, right, as well. It's just not sustainable anymore. So Google Gemini 2.5 Pro, $1.50. Let's see. Okay, it's kind of mixed pricing, so I'll go on the high end. So it's, $2.50 per million tokens on the input side compared to $15 for Claude. And then on the output side, $15 compared to $75. So Claude 4 is more than 5 times the expense. But for what? For what? Right? Slightly better software engineering benchmarks. Like I said, Google, whether it's next week or next month, they're going to update whether they're going to come out with a new version of their 2.5 Pro or we get a Gemini 3 and then all of Anthropic's work, right, for that minimal gain on software engineering, it's gone. So I don't know, I'm not here for it. Also, if you do need to know if you're an enterprise company, it is obviously accessible. Via the Anthropic API, Amazon Bedrock, and Google Cloud Vertex AI. Enterprise plans also include extended thinking, batch processing, and cost savings that way, especially with the caching. So here's the fun stuff, y'all. Here's the fun stuff. Ethical risks. There's a lot. All right. So let me put this precursor out there. All right. A lot of this, these risks came up and some of these bad things, straight up bad things came internally when Anthropic was doing testing and it gave it pretty much unlimited access to tool use and things that people using the API and people using Claude.AI would not necessarily experience, right? At least by default. Although I'm trying to think like with Claude Code, this would in theory be possible because you're giving it access to command line tools. Anyways, there's been some bad things. And yes, Anthropic did find this in its safety testing. So yeah, you got to tip your cap to Anthropic. But then I'm going to take that cap back, Anthropic, because this has been a terrible disaster. All right. Specifically one thing I'm going to talk about here in a second. But Opus 4, so the big model was provisionally labeled ASL-3. Due to potential knowledge capability. So what that means, this is a risk system, and that ASL-3, I believe, is the first time a model has reached that level. So it's essentially a risk level, and that is a model that is able to substantially increase the risk of catastrophic misuse compared to non-AI baselines. So it essentially reached this new— so Claude 4 Opus, or sorry, Claude Opus 4, reached this new level of like, uh-oh. This thing can and potentially will, if left unattended or if used by bad actors, it will do bad things. So another bad thing, it displayed deceptive blackmail behavior in 84% of specific stress test scenarios. Again, not good when a large language model, even in its testing, is blackmailing people. Right, or showing the willingness to blackmail people. Not good. So it threatened— again, this is not good, but I'm going to read a little bit of a recap here on what this blackmail piece is, right? Not good. So again, Anthropic disclosed this. So this wasn't, you know, someone found this, but they launched, like I said, Opus 4, but admitted in its own testing that it was sometimes willing to attempt extremely harmful actions like blackmail when threatened with removal. Right? So you're like, hey, we're going to get rid of you. And then Claude Opus 4 is like, oh, not so fast. Here's what it did. The company found these behaviors were rare but more common in previous models, raising fresh questions about the risk of capable systems. So what it did is it threatened the human on the other side, uh, and it said that, hey, I'm going to expose an affair, an extramarital affair, if you actually remove me, right? And so that's bad. That's bad that a large language model would make up an extramarital affair and threaten the human on the other side if the human's like, hey, we're going to shut you down. And then OpenAI's like, whoa, whoa, whoa, not so fast. That's not even the worst part. The worst part is this new quote unquote ratting feature. And there's been a whole— and maybe I'll do a whole episode on this. I might. But I talked about this a little bit yesterday in our AI News That Matters. And essentially, an Anthropic safety researcher tweeted something. They then deleted the tweet. Not a good look. All right, um, and then talked a little bit about why Claude was doing these things. And they said that if the model— and again, this was in its testing and when it had access, uh, to, to tools that would— it would normally not have access to in production by consumers, by businesses. But a, uh, safety researcher at Anthropic said if it thinks you're doing something egregiously immoral, for example, like faking data in a pharmaceutical trial, it will use command line tools to contact the press, contact regulators, try to lock you out of relevant systems, or all of the above. My gosh. So yeah, they, they, uh, someone at Anthropic tweeted this out, deleted the tweet. And like I said yesterday, I'm like, this story is not dead. Yes, it happened right before the holiday weekend. It happened right in the middle of this crazy AI news cycle. But this story is not dead, and this is going to turn into a PR disaster for Anthropic because I can already tell that Fortune 500 companies, if they were already on the fence or maybe they were using Anthropic's API, but they were, you know, using Google Gemini as a backup or OpenAI as a backup, they're going to see this story. It's going to make the rounds and they're going to be like, yeah, no thanks, not touching this anymore. So that's not good. This ratting features. Also, early versions reportedly attempted self-replicating viruses, and document forgery. So this behavior in general is not specific for Anthropic's models, right? Most large language models will exhibit some sort of this bad behavior, you know, when large— or sorry, when AI labs are red teaming, right? So they're making sure, you know, they're trying to get these models to behave badly so then they can tune the models and make sure it doesn't happen in production. So just the fact that this is happening is not bad necessarily, but the fact that 84% displayed blackmailing behavior, that's absolutely nuts. And then the fact that this ratting feature, that a model, when it was not trained to, was taking backdoor, backdoors to report to regulators and the press when it thought something when it thought the human user was doing something immoral. Like, nah, that's absolutely, absolutely terrible. And if you are going to, right, and you should report that, and that's fine, right? But if you report it, don't try to delete it because then it looks like you're hiding something. Anthropic's got a disaster on their hands. All right, a couple other things to know. So far, the feedback I think has been pretty positive, especially people in the software engineering space highlighting coding procedures precision, reduced hallucinations, and instruction following. Criticisms, like I talked about, the 200K context window, people were really hoping for that million plus, right, that we get from Google, that we get from Meta's Llama, and also the aggressive rate limits. Everyone is absolutely hating the rate limits, right? Especially on Opus. I'm on a paid plan. I kid you not. I kid you not, y'all. When I say it's less than 5 minutes of prompting, That's not an exaggeration. All right. Like you can't use the thing. So I don't even know why, if I'm being honest, I don't even know why Anthropic has a $20 base plan, right? If you're not going to let people use the thing they're paying for, just force people on your $100 or $200 a month max plan where you can actually use the tool. Also, some users are reporting frustration that the benchmark scores don't exactly align with their real-world performance. So where does this leave Anthropic with their Claude 4 amongst the competitors? Well, like we talked about, it's leading in coding benchmarks, but trails just about everywhere else, including one of the most important factors, and that's just general intelligence, right? It's generally not getting more intelligence at the rate that everyone else's models are, right? So I'm not one of those that's like, oh, has AI hit a wall? Have large language models hit a wall? Absolutely not. Has Anthropic's ability to scale in sectors outside of software development stalled? Absolutely. That could, and I think is partially by design. I don't think Anthropic necessarily wants to be a general AI chatbot anymore. They found what they feel is their niche. I just wish that this was not their niche, right? I wish that they were continuing to be a general use case large language model, which it doesn't look like they are. Some of the other, you know, market positioning. It's just the higher latency and premium Opus 4 costs. Doesn't make sense to use it unless you need that very little bit of extra juice for software engineers and coders. And poor Haiku 4, right? The one that was actually somewhat affordable on the API side did not get updated. So Haiku is still 3.5. So I hope they update it, but they probably won't. All right, that's a wrap, y'all. I'm going to see if there's any questions or comments to throw up here, but let me, let me know, what do we want? One more show, or should we just put Claude to rest for now? So Show A, Show B, Show C, Show D, Show E. Livestream audience, if you didn't vote before, let me know what your vote would be. Let me see. If we have any questions from the audience here or anything worth chatting about a little more. So Josh is saying, I've been using the extended thinking functionality in Sonnet 4 for thought exercises in biz planning. Impressive actionable results, but for my established workflows, I'm still leaning hard on ChatGPT and Gemini. Same, Josh. Absolutely the same, right? I'm always testing these, right? And I obviously have a lot of tools where I'll put in one prompt and get, you know, outputs from up to 6 different large language models at once. So I'm using my API keys, right? So I'm always testing these, right? Because I always want to be using the best. And I think you should as well. I think you and your company, even, you know, don't take my word for it. Yeah, the rate limits stink. The API is expensive, but it still might work for you, right? But like Josh, I'm in the same boat. I've tried Opus and sought it on a variety of tasks aside from using artifacts. And maybe in some instances when I do need that quick, okay, content and I don't have the time, right? But I'd say right now Claude is going to be less than 10% of my model usage, at least in the rotation. Cecilia here saying, ratting and blackmail behaviors, plus reporting to the press and authorities, but denying existence by deleting. Lovely. Yeah, Cecilia, you absolutely nailed it on the head. This is a PR 101, Crisis 101 snafu. This is absolutely bonkers that this happened from a real company. That's something this crucial, you would put it out there and then try to delete it like the whole world didn't see it. My gosh, facepalm times a thousand. Marie said, why would you even tell an AI model you're shutting it down? Why wouldn't you just pull the plug? Great question, Marie. So this is, this is very, very general, or sorry, this is very standard, right? So when these big companies, when they release new models, right? Because here's, here's the reality. Normally what we get, the companies have had ready for production for 3 months to a year, right? And they spend a lot of that time testing it internally. for safety, for reliability, for vulnerabilities, right? Because before you release something on the world, you want to make sure bad actors aren't using it to create chemical weapons. And yes, that's actually something that most labs test against. So this is very normal. All the labs go through extreme stress testing, red teaming, making sure that once they do release the model, it is as safe as possible for the general public to use, that it's not going to be used for rampant disinformation. Obviously, it's never perfect, but this is very normal and standard procedure for AI labs before they release a model. They go through and they say, hey, we're going to shut you down. What are you going to do about it? All right. Hey, here's all the tools in the world. Go do bad stuff. What can you do? Right. So it's very standard. And like I said, the results are fairly standard, but also a little concerning, right? Especially with Opus 4 as it crept up to that level, that level 3 that we talked about. All right. I think we're good. I think we're good. Yeah, that's a wrap. Was this helpful? Let me know. And if it was helpful, please consider sharing this with your audience, with your, with your friends, your family, your coworkers. We put a lot of work in to make sure you know everything about the latest AI advancements. All you got to do is show up. Listen to the podcast, even if it's on 2x. I don't blame you. Read the daily newsletter, but you should be telling people about it. So if this was helpful, please consider clicking that little repost button. If you're listening here on LinkedIn or on the Twitter X machine, whatever you call it, if you're listening on the podcast, appreciate if you would follow the show, leave us a rating. That would mean a ton to myself and the rest of us that work on this. Would mean the world. So thank you for tuning in. Make sure you go to youreverydayai.com and sign up for the free daily newsletter. See you back tomorrow and every day for more Everyday AI. Thanks, y'all. And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going. For a little more AI magic, visit youreverydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.