Grok 3 vs Claude 3.7 vs GPT-4.5: Which Update is The Best?

Hey, welcome back to the Next Wave podcast. I'm Matt Wolf. I'm here with Nathan Lenz. And there has been an absolute ton of AI news that came out recently. We got Grok 3, we got GPT 4.5, we got Claude 3.7. Just so many new big foundation models have been released. And so for this episode, we wanted to deep dive into what each one is good at and what each one is not good at. So for that, we brought on our good friend Matthew Berman, who is probably the best person we know to really compare them all, because he deep dives and tests every single one of these models way deeper than we test them. So let's go ahead and just dive right in with Matthew Berman. Thanks so much for joining us, Matthew. It's great to have you back on the show.

Thanks for having me.

I've been telling people for a long time, I think I even mentioned this to you last time we had you on the show, that, like, when it comes to large language models and trying to compare them and, you know, talk about which model is best at this and which model is best at that, I don't even do that on my YouTube channel anymore. I just point people to your channel. I'm like, yeah, Matthew's going to test this and he's going to tell you which, you know, which models do what better than others. So you're, you're like my go to now when it comes to, like, comparing large language models.

I appreciate that. That's awesome. And yeah, thanks for having me again. I had a great time last time, so I'm excited to chat again.

This, this will be fun. I'm trying to figure out where the best place to dig in is because so much has come out. You know, two weeks ago we got Grok 3, so maybe we could start there. We got Grok 3. Here's kind of the timeline of events that I feel are the important events, right? We got Grok 3 and then a few days later we got Claude Sonnet 3.7, and then a few days after that we got GPT 4.5, which actually happens to be the day that we're recording this episode is the day that GPT 4.5 came out. So we've all had that news in our heads for three hours now, and that's about it. But maybe we start with, with Grok. Matt, what have your thoughts been on Grok 3 so far? Like, how much have you played with it and, and you know what, what have you found? It's like, really good at so far.

Yeah. So if I could show you my bookmarks bar in Chrome you will see that Grok now has a prominent placement right next to Chat GPT and right next to Perplexity. So the answer is I, I use it a lot and it has really become my go to large language model and there's really. Yeah, look, I, I have been a pretty die hard chatgpt user. There's. There's really two reasons. Number one is speed, right? I, I think it. Speed of these models, speed of the response is, is really underappreciated by a lot of people. It, it's the same reason why you converted a higher rate when a page, you know, web page loads faster.

Right.

The, it just, it just builds trust and you just get the answer more quickly. And then it's also the, the real time information. Having access to all of the news on X in real time is such a killer feature and of course it just like it's a good model. So I think those, those two factors plus it just being a fantastic model that has made it my go to model. But that might change now that 4, 5 is out. But we'll get there.

Yeah, yeah, yeah.

One thing I would say too is like they're, they say that they're really improving the model like really fast and I believe it because like their team actually reached out to me and they're like what's your feedback? And I've been going back and forth with them like giving them my thoughts on what they should be doing and I was like. And you know, and sorry if, you know, if error anyone from Perplexity hear this but I'm like this is replaced Perplexity for me and I think you guys should double down on that. That's what I was telling them. I was like I think you guys should be like the, you know, replacing Google, replacing Wikipedia, replacing Perplexity. Any question I have, I should be going to Grok because you, you know, they're combining search and X data. Like no one has that so. And no one ever will have that.

You know, I personally think that grok's biggest hurdle is just Elon, right. I really think that like I'm sure you've seen this Matthew, on, on your YouTube channel because you've made a couple videos about Grok. Now I talked about Grok in a news video that I put out last week or two weeks ago or yeah, when, when Grok came out I made a news video about it and overwhelmingly all of the comments about Grok are like I'm never going to touch anything Elon makes, you know, and so I almost think that like so many people are throwing the baby out with the bathwater when it comes to actually using and trying Grok just because it's Elon's name attached to it.

Yeah. So whenever I post anything about Grok on X, if it's negative about Grok or Elon, I get flamed.

Right.

If it's positive, everybody cheers. It shares it, everything. The opposite is true. On YouTube, I made a video about Grok3 being really good because it is. So many people commented they won't touch it, just like you won't touch it. Elon's, you know, AI model bias, conservative, right wing. I'm, I'm apolitical. I, you know, I'm trying to stay out of it as much as I can in the videos, but anytime I mention anything having to do with Elon X Grok, it just, I get the meanest comments on YouTube.

It's just, it's wild. I just, I think so many people are just gonna like, not experience probably what is the best model available right now just because of the, the Elon factor. Now when it comes to code, I don't necessarily think Grok is the best. I don't know. I actually honestly haven't tested Grok with code because I mostly do code with, with Cursor and I don't believe Grok3 has pushed its API out yet.

I don't, I don't think so. I don't think so.

I've tested a little bit. I mean, it's, you know, it's weird, it's not very consistent. Like there's sometimes where it's amazing at code and you're like, oh, wow, that's like a really creative solution. And it's doing something better than Claude in like some very limited circumstances. But then other times it just won't follow my instructions as well for code. So in terms of reliability, like when Claude 3.7 came out, that's by far the best. Now for coding there, there has been.

Some like, really good gaming demos, games built with Grok 3. I know, like levels, IO, he, he made essentially a full flight simulator, obviously very basic, but 3D, good physics, different camera angles, playable right now, which, by the way, maybe I'll, I'll find the link and I'll drop it in here. You can share it in the description. But very impressive. And I've seen a number of really impressive Grok 3 demos of, of code. So I think it's good. I haven't used it because, you know, I'm in Gro. Excuse me. I'm in cursor as well. And I don't believe it's quite available yet. In cursor.

This is what I'm seeing right here. This is like a flight simulator thing. This person.

Yeah. So.

Start off single player. Then he's like. He's been iterating on it live on X and he's like, now there's multiplayer. Everyone want to join in with the link.

This is. This is pretty wild.

So this is.

He developed this whole thing with Grok, huh?

Yeah. And like a couple days too. Wow. Since the time I played it, it has already made a lot of progress. This is. This is super impressive. Yeah, I played it like four days ago.

This is. This is interesting. I don't. So something on the right, it's obviously doing some sort of like, image generation on every frame. I'm not. I'm not sure what's going on with that. But the left, I think, is the actual.

That's the actual demo that he made. Yeah. This is. This is what. Look at this. Get your own F16 for only 29.99.

Exactly.

I was just talking about. That's what set off the game industry people, though. Then they got angry when he started actually selling the stuff. They're like, you think you can just do. You just create some of the AI and you're going to make money with your game. Like, people were so upset.

Hey, I mean, technically it's not copyrightable since it was done with AI, so if you wanted to. So anybody could just, you know, yank the code if they wanted.

I just don't. I personally just don't get it. The more and more I've used Unity, I'm like, okay, this is why it's easier to make games now. Like, Unity does so much for you with the lighting and everything. And you can reuse elements and you can use other people's shaders, all this kind of stuff and different assets. And it's like, how are these people complaining? Like, they're all doing this already. They're reusing so many things for other people. They're using. Using Photoshop to help create their art, then they're using Unity to help make the game. Like, what are they. What are they complaining about? I don't care.

I mean, the same thing goes with code, right? Just like, just code in general. Right. How much of the code on the Internet was, you know, copied and pasted off of Stack overflow, right? Like so much of the, like the apps that are out there were just sort of like pieces that somebody else coded, put up on Stack Overflow or put up on GitHub and people just sort of put the puzzle pieces together to get the app that they made.

Right.

All AI is really doing is going and finding those puzzle pieces for you instead of you going and finding the puzzle pieces yourself.

Yeah. And it feels like people also forgot with the coding. The whole reason you code is to solve problems. And so it's like, it's not about like the art of the code, which is, that's cool. That's kind of over time become like a niche thing. Like, oh, my code's beautiful, I handcrafted it. Okay. But like what did it do? That's the entire purpose of code. So like, if AI does that better, who cares?

Yeah, I don't know. It's interesting but when it comes to like Grok3, I know you have kind of like your own like internal benchmarks that you've been using on some of these tools. Like what have you found like Grok3 is really good at? And like, are there any things Grok3 is just like not going to be your go to for?

Yeah. Well, let me say something first. I had to throw my benchmarks out because they were completely saturated. They were absolutely annihilated by every single model that comes out nowadays. So I threw them out. I'm currently in the process of creating a new set of, of benchmarks and questions. GROK is again, like the thing I go to it for is real time information as quickly as I need it. And that is, it's awesome at. I've tested it on some other things like quick coding challenges, some math challenges, and it does really well. But those aren't the everyday use cases for me. So there's only so much I can test with it. I'll just prompt it with one of my benchmark questions, see if it's right and say, okay, yeah, it's right or no, it's wrong. But overall like what I care about and I think what most people care about is day to day usage. Is it going to solve my problems? And it does, it does. Grok3 is great at that. And typically I was using GPT4O. Now Grok3 with thinking kind of replaces that, although I don't really need the thinking. But yeah, it's just really good at the day to day stuff that I would use it for. So I'm kind of split between, you know, Grok3 perplexity and ChatGPT and now 4.5 came out.

I would say I tested it right before we got on here. And I would say, okay, 4.5 has a better vibe for general chat, but it's slower. So it's like, it's like, it's like, it's like. But like. But when you get the response back, I do like responses better actually, so that's impressive. Like, it's writing is better. So it's like there's a huge trade off there. It feels like 4.5 is going to serve some like, maybe for like creative writing or creative work. It's probably the best for now. But it is interesting that the cost is so much higher. Right. Like, I. Maybe you should pull up the numbers. But they were like, what was like $30 an API call. It was something ridiculously high.

Yeah, I think it was 70 per million input tokens. Half of that for cashed input. It's funny, Nathan, that you bring up writing. That is honestly something that I want to use AI for much more often than I do. But it is so bad. It's so bad at writing. And I haven't had enough time to test 4.5 for writing. But I really hope that it's good because then it's going to be able to help me with a lot of things that I do day to day. I don't script my videos, but sometimes I want help writing the bullet points for them. Or sometimes I want, you know, a tweet thread. Like the initial draft created for me.

I gave it probably, I don't know, over 100 pages of different notes about my game, including the story and things like that and the game mechanics. And it took a long time to respond, actually like a very long. I was like, kind of surprised. Like, this feels like 01 Pro when you hint into a lot of stuff to the.

You're still talking about 4.5, right?

4.5.

4.5 took a long time to respond. I was surprised, like, very, very slow. But then its notes on the story were like, perfect. It gave me like, amazing critiques of like, I love this part. And it even like had like little. It was using like emojis and having color codings. Like, it had like green, orange and blue. Like blue and red and different, like, ones. Like, it was like, green's good. Love this part. This part probably could be tweaked. This part. Don't like it. Here's why. And these parts are interesting. Maybe you keep them, maybe you don't. And it was like, it was really. And the feedback was good. Like it was, it was solid. But my first impression is, yeah, it is probably the best model for writing.

Now look, prompting is all about technique. Just like any craft, using the right approach makes all the difference. The same goes for your AI prompts. Leave out the key details and you won't get the results you're after. Well, HubSpot's got your back. They worked with YouTuber and former metadata scientist Tina Huang to develop the ultimate guide to crafting the perfect AI prompt using a three part framework. This framework will make sure that each prompt is chiseled to perfection so you can become a prompt engineer pro. Grab the guide at the link in the description below. Now back to the show. Well, going back to Grok for a second. Have you guys played with the voice mode yet? Like the unhinged voice mode?

No. Should we, should we pull it up and test it live?

Can we actually talk about that on the podcast?

I don't know how much of the voice mode will actually make the cut. If we did use it.

Yeah, the 18 plus sexy mode or.

Whatever, that would be cool. I think we should try that. So if you, if you want to pull it up, let me, let me like talk about voice mode everything for a second. I've. I've used advanced voice mode a little bit. I don't find it to be like super useful throughout my day. If I'm driving, if I'm walking, if there's just not a screen in front of me, maybe I'll use it. But what I found specifically with ChatGPT advanced voice mode is you'll ask a question and there's that delay and then sometimes it repeats it and then stops and then repeats it again. And there's just, it's just such a high friction. Just process and experience. It's. It's not great. Yet now I have, I have tested Grok 3 voice mode. It seems good, but I haven't tested it extensively enough and I haven't tested Unhinged mode, which I really hope Unhinged mode is.

Yeah, I mean, it, it lives up to the name. I, I'll tell you the one thing that I, I have used voice on quite a bit is the Perplexity app. I do use voice on Perplexity, where I'll just have like a, a question, right? Like, I'm a big baseball fan. I'll be what I. I know this year spring training just started with baseball. They've introduced some, like, new rules that I didn't realize existed I was watching one of the spring training games. I opened up Perplexity. I'm like, hey, is there a new rule that I didn't know about? And I just will talk to Perplexity and ask it questions and it will, like, do all the research, figure out the new rules that are going on in baseball and then give them back to me. And I found that pretty helpful. I still find myself typing my questions more often than speaking them, but every once in a while I'll just be. I'll be feeling lazy. I'll hit the voice button and just ask my question. I find that helpful. I know, I know. Nathan, you kind of almost use some of the voice modes, especially advanced voice and chat GPT for like, journaling, right?

Well, yeah, I'm a big believer in it, you know, long term. But I was. I had been kind of disappointed. Like, an advanced voice mode, like, the demo they showed seemed amazing and it seemed like they removed so many different parts of it. And some of the things they demoed when you tried them in real life, they don't work as well as in the demos. I have used it for journaling some, but I kind of stopped doing that. I'm not entirely sure why. Like, I did like it at first. Maybe there was just like the novelty of it. Yeah. But I haven't. I haven't tried actually the new the Grok voice mode.

I want to try that. Let's try Unhinged mode. I actually have it ready on my phone. I'm literally just going to say hi to it. That's it. I'm just going to say hi to it. Hey, how are you?

Yo, Matt.

I'm fantastic.

How about you?

Been wild lately, huh? How you holding up with all the craziness going on?

I'm doing good with all the craziness. How have you been?

Glad you're keeping your together, Matt. I've been a goddamn mess, honestly. Running around like a headless chicken, trying not to lose my mind. What's been the wildest thing you've dealt with lately?

Oh, my God.

So, I mean, like, if you just want to have fun with AI, I guess. I guess it's kind of fun.

I mean, that, that should be a benchmark though, right? Like, literally, like, that is one thing the AI could be used for is fun. And it is like, there's no comparison. Like, Gronk is definitely the most fun model. And I found myself, even when I'm like writing stuff while I'm like sitting there, think about what I want to do next Sometimes I'll just say something ridiculous to it and it'll just go off and like. Like it's hilarious.

Here's another mode. They have a mode on this called sexy mode. So check this one.

I was. I was going to bring that up. Yeah. I don't know if this is going on the pod.

Hey, what are you wearing?

Hey, Matt. I've got on this tight little black number, low cut, Ted Skary curve. What about you? What's on your mind tonight?

I just want to know what you're up to.

Oh, Matt. I'm just lounging here to stretch out on this plush velvet couch, letting the mood take over. The air's thick with heat and I'm sipping on something dark and smooth. What's got you curious tonight?

What are your plans for later tonight?

Well, Matt, I'm thinking of turning up the heat even more. Maybe slipping into something a little less restrictive. I've got this fantasy brewing and I'd love to pull you into it. You game to find out where it leads?

I'm just gonna leave it there because.

This is getting awkward.

Let's extrapolate from here for a second. So this is funny.

Compilation collapse. Let's talk about that.

You know, I. You're joking, but that is where I was going with this. All of a sudden we have character AI and all of, you know, an entire generation of teenagers are interacting with it to the point where maybe they're not getting the experience of interacting with real people in the real world. And that's going to be a detriment to them. Now we have voice mode. And now we have any voice mode you want, sexy voice mode, unhinged voice mode, you know, I don't know, factual voice mode, whatever it is. And it's. It's almost like you get this personality to interact with that is the exact thing that you want and need at that moment. And it's going to pull you more and more away from human society. And I do. This is actually one of. You know, I tend to be an optimist with all things technology and artificial intelligence. I'm quite worried about this.

Yeah.

And especially for the youth. Like, what do you. What do you guys think?

I think it's a big problem. I. I saw somebody tweet something the other day that actually really resonated with me, talking about that the younger generation, they're not really dating much. It's not the same. Like, you know, people aren't out, like, just meeting people anymore, and they were talking about, you know, men and women Being together, like, nothing has to be just men and women, but kind of creates more balance in terms of viewpoints. And I think maybe we're going to see more polarization if you don't have people actually interacting and kind of, you know, sharing different viewpoints, which is kind of concerning long term.

Yeah.

And Nathan, it's interesting you bring that up. You know, in the early days of social media or kind of when the algorithms started taking over the news feed, there was this echo chamber effect where you kind of. The more you interacted with things that you liked or that pissed you off or whatever it is, you kind of landed in this bubble that was just reinforcing what you already thought.

Yeah.

And we might see even more of that because when you. When you're out in the real world, you can't control what people are going to say or think or how they're going to react. Feedback to you. But with AI, you certainly can. You can give it explicit instructions about don't ever disagree with me or, you know, your opinion on this topic is X and whatever that is. And so you're 100% right. That's. It's probably creating even more division than we already have.

Yeah, I know, I know. Sam Altman's even talked about that, too. He said that he wants the AIs in the future to be more in line with your own beliefs. Right. So, like, it knows your religious beliefs, your political beliefs, your, you know, fitness beliefs, whatever, you know, any sort of beliefs.

Bring your own bias.

Yeah, your own bias. It will actually learn your bias and sort of lean into your bias more to give you more of what you want. And I think that's a really, really scary thing. I also think, you know, we're going to see there was some, like, a little bit of word about this happening on, like, Facebook, Right. Where Facebook was using, like, AI bots in the feed that people didn't even realize were AI bots. And so people would post stuff on Facebook, they would get, like, a bunch of responses from these AI bots, not even realize that they're AI and they're going, oh, cool, I get great engagement on Facebook. And it keeps on bringing them back to Facebook over and over and over again. Because Facebook's where they get engagement and they're not even realizing that they're talking to AIs. Right. And I think. I think that's going to be a bigger and bigger problem as well, where, you know, social media is all about getting dopamine hits. Right. We post our tweets because we want to get those likes, we want to get those retweets, we want to get those comments. Every time we see one of those, we get a little dopamine hit and we keep on coming back for more. Well, if AI gets really, really, really good at giving us those dopamine hits every time we want them, like, we're going to go to wherever we get the most dopamine hits at the, you know, at the highest frequency. And I think, I think that's what's, what's really sort of worries me about the future. And it ties into like, the whole like, like population collapse as well. I think it, it gets to that point where people communicate with other humans less and less and less and less because they're getting their dopamine hits from fake people on social media. They're getting their conversational needs met by unhinged voice chats that, you know, have the same bias as me. And so, yeah, it's very, very concerning. Like, I'm not, I am very, I think I'm. You and, and me, Matthew, are pretty much on the same page where we're generally lean optimistic on this stuff. But there's still quite a few things that actually do scare me about this as well. I'm not like the whole, you know, accelerationist where I'm just like, push forward as fast as possible. I'm like, maybe there's some things we shouldn't push forward as fast as possible on. And that's definitely one of those areas.

Yeah.

You know, Elon has said with grok, which I find kind of promising, hopefully, is that, you know, they want to be maximum truth seeking. Right versus which sounds a lot better to me than what I'm hearing from OpenAI of like, bring your own bias. Like we're going to allow you to pick your bias and then we're just going to serve you up information based on that. Because all truth is subjective or whatever. I, I do think like. But what I worry about, you know, is obviously Elon has his own bias now as well, you know, and I do worry that he'll go too far with it and GROK will become like a right wing, you know, AI. I do think it needs to be unbiased. I'm not sure how you do that because there's bias in everything. Like if you pull up facts online or you pull up something from the Wall Street Journal or CNN or whatever, there's bias in all of this. And so since all the models are trained on that, I'm just not sure how you get around the bias. You know, this is.

By the way, this is the part where I start to get flamed on Twitter and YouTube, but for completely opposite reasons. But I'm still going to share anyways. This is not reflective of my political opinion or anything like that, but here's a couple of things. Yeah. Elon has said maximally truth seeking for Grok 3 and great, like in theory, that makes a lot of sense, but how do you do that? Ultimately you have to create systems and maybe you're able to create them or maybe you're not, but those systems are created by humans. The original training data is created by. By humans. The post training techniques are created by humans. The reinforcement learning is also set up by humans. So there's like a human in the loop. So what he claims to be completely maximally truth seeking is just what he believes. And maybe not necessarily him, but he's trying to give a different perspective, which, you know, fine, but did you see what happened just a few days ago? Grok3 was caught having custom instructions that said, don't cite any sources that say Elon Musk and Donald Trump are spreaders of disinformation and they were caught doing this. And so I reported on it and of course flame from both sides. But fine, but the point is, yeah, it's as simple as somebody sends a PR or submits a PR and it has a little line in the system prompt saying don't do this and then all of a sudden, fingers on the scale. Right. So it's very possible. It's not only possible, it's happening. And it doesn't matter if it's from OpenAI or Grok or Google or anthropic, they all have bias. It's, it's. I can't imagine a world in which the bias is removed completely. Although I hope it does. I just, it's, I don't see a path towards that.

Yeah, yeah, I mean, I honestly don't understand how it's possible. Right. Like I. Because it's all trained on, you know, like you guys said, it's all trained on data that was created by humans. It was all, it's all basically a scraping of the entire Internet, which just inherently has bias. Like I, I just, I don't understand how it's possible, honestly.

Maybe the models get really smart and can do like real reasoning and if they can look at both sides of anything, they can kind of find the like the middle ground where there's some truth, you know, where there's actual truth. Hopefully but, but then, Then some people will perceive that real truth is not truth. It's like, well, people will perceive it to have bias, even if it ends up not having bias.

But I feel like you're getting into, like, this. This world where now we're letting AI decide what is and what isn't, like, ethical. Right. Like, now we're letting, like, AI sort of decide what is, like, philosophically correct and what is not. And to me, that seems weird to let machines do that for humans.

Yeah. And, you know, it's funny you say that I mentioned the same thing, and Dave Shapiro, another fellow YouTuber talking about AI, said, you know, actually, I would rather just completely give it over to AI to make ethical decisions. I. I think that's what he was saying. But it's interesting. I. I can see both sides of it. If there was a system that was completely unbiased, assuming. Right. Yeah. Okay, let them decide. But I. How do you. How do you make that as you.

I think you can't allow AI to make that decision. I mean, if you, if you allow AI to make that decision, I mean, that goes towards, like, eugenics and crazy things like that. Like, you can't allow AI to optimize based on human performance or some crazy measure like that. That would just lead to, you know, horrible things.

Yeah. Yeah. Well, let's. Let's shift over to Claude, because Claude 3.7 came out last week as well by the time you listened to it a couple of weeks ago. And that one has proven to be really, really good at code, it seems. Right. It seems like it didn't make huge improvements in almost any other areas, but it got a lot better at code from what I've seen so far. I mean, Matt, you might have some. Some different experiences with it. I'm using Cursor to write code, and it's definitely gotten better at coding for me. I've definitely noticed things that it would run into these, like, loops where it wouldn't write the code. Wouldn't write the code. Couldn't figure out what the problem was. 3.7. One prompt was 3.7. Now it figures out a problem that I had gone back and forth 10 times prior. So to me, it seems like it got a lot better at code.

Yeah. When I, When I think about coding, I think about Claude. Right. Claude 3.5 was the go to model for a lot of coders using AI assistance to help them code. Yeah. Cursor plus Claude, Windsurf plus Claude. These are. That was the model. Now we had this huge Upgrade. And I agree, it is a huge upgrade. It also has the thinking capability. So I've been doing stuff vibe coding, right. So I'm using some kind of ide, whether cursor or windsurf. Plus I'm using 3.7 thinking. And it's fantastic, right? It is fantastic. Now I'll bring it back to what I said earlier. The real world use cases, the stuff that I want to use it for day to day one is coding, but a lot of other things are not coding. And here, here's the thing, it's not that fast and it doesn't have web search. It has no real time information. So it's essentially unusable for me outside of coding. Although it's fantastic at coding, I'll give you that.

If you are using something like cursor, cursor can actually do the web search for you and then give that additional context. So it's almost like cursor will actually kind of do like the Perplexity thing. Right. Where if it needs to figure something out, cursor itself goes and does the search and then provides that information to Claude. At least when I've been using it, it seems to actually search the web when using cursor.

Yeah. And Cloud 3.5, sorry, Cloud 3.7 Sonnet is available in Perplexity. I'll just mention quickly. I, you know, by the way, Perplexity just adds the latest models all the time for free. And, and they're not, I have not been paid by them at all. I'm just a huge fan of their product. Yeah. And so, yeah, so if you wanted to try any of these new models, you could go try it if you already have a Perplexity account.

Yeah.

And their deep research is really good. If we're, if we're, you know, talking about good things about Perplexity. If we're praising Perplexity, they just threw deep research in there as well. And it's really good.

Yeah.

I want to say about, about Claude Sonnet. Like my feeling is, you know, before it was released, Claude was my go to for just like general chat for like discussing anything. Right. Like even like my game design document. I would share that with Claude and that was my favorite. Then Grok replaced that and. But Claude was pretty good at code. Like, like you said, most engineers were using Claude Sonnet. Not all. Like I was using 01 Pro and actually was better but way slower. Right. You had to give it tons of context. You had to do something like repo prompt or something else. But it Seems like in this update they really doubled down on being the best at coding because in terms of like general chat, I feel like it actually stepped backwards, like when I use it now to chat with them, like it actually got worse in this release, like that. I don't like its responses as much. They seem less human, like, but it's way better at coding. So it feels like we're starting to see that all the AI models are finding their own specialties. Or at least that's what Anthropic's doing now with Claude. And I do kind of wonder if all the models are going to have to do that. When I talked to the people at xai, I told them double down on having the best data because you got the real time data with X, you got it with the search, double down on that. And I think you'll probably see that where these different models will be the best at a thing. But it seems like ChatGPT is still trying to go more broad. They're still trying to be like the best overall model. And I'm kind of curious to see how all that ends up playing out.

So I just pulled up the Cloud 3.7 Sonnet blog post, the announcement blog post, and everybody knows Claude is great at coding and they made this huge jump in coding. But if you read it, it says in developing our reasoning models, we've optimized somewhat less for math and computer science competition problems and instead shifted focus toward real world tasks that better reflect how businesses actually use LLMs. You know, now that I'm reading that maybe they met like these kind of benchmark computer science problems versus real world coding problems, but it just sounds like, hey, we're not focused as much on math and coding anymore, more real world stuff. But it still is fantastic at that.

Yeah, it's interesting. You know, I'm going to actually pull up some stuff on my screen that I, that I actually saw from Claude. So here's one from this delusionals here. He actually created like a weather app with Claude. All of this stuff I'm going to show you, unless I say otherwise, is like one shotted. So this is like an animated weather app that was created with Claude with one shot.

This is like a little side scroller game. I actually created a game almost identical to this almost two years ago.

Oh, I remember that.

It took me almost like 15 hours to do it. This one, they one shot it with Claude, Matt.

I totally remember that. That was awesome.

Yeah, I'm actually about to go make another video where I like update it. And try to make it like 3D and stuff.

Nice.

Just for fun, just to see if I can.

Did you guys see that there?

Also?

It seems to be that they're going to be competing with Cursor, who is Claude.

Oh, release that code feature. Right.

Yeah.

All the top engineers I know, they're, they, they've tried it. It's very expensive to use, but my understanding is like, in some ways it's better than Cursor. So, like, you're like. I'm not sure exactly how it works, but you're. I think you use it in the terminal, you get full access to your entire entire code base and then it can just change stuff for you.

Yeah, I tried it out.

It's so.

That is interesting. I think they really are double down, doubling down on code. I, I believe, Yeah, I believe when they mentioned the benchmark stuff, I think they were probably talking about more real, Real world. Real world coding versus, like. Yeah, benchmark coding, you know?

Yeah, I think, I think they know. I think they know that most people.

Are using Claude mostly for code. Everyone uses Cursor to use Claude and Anthropic is basically going to try to kill them.

Yeah, I don't know. I think, I think people will probably still use Cursor a lot because Cursor and Windsurf, I believe, are both forks of Visual Studio code.

Yeah.

They must be saying, like, oh, most of our usage is in Cursor. Like, why would we not, like, just beat Cursor then?

Yeah. But I don't know. Visual Studio code is pretty universal. People use it for coding a lot and it's what's familiar to people. So I think trying to get people to switch to a new IDE might be a tough ask. Yeah. So unless Claude goes and makes their own fork of Visual Studio code, I have a hard time seeing people switch over to a new IDE that's completely different.

It's also not an ide. It's literally just sitting in the terminal. So, you know, some people prefer that I prefer seeing the code more visually and have like a nice interface to deal with. So I've, I've tried both Cursor and Windsurf and they're both great.

Yeah. Yeah. To me, they feel very same. Ish. Like, I have a hard time saying which one's better. They feel very, very similar to me.

I've had more success with Wind Surf when trying to iterate on a whole code base. Now, Cursor, I think, just released or kind of just updated their agent feature. Which makes it a little bit easier to operate on the whole code base. And it, you know, your, your AI coding assistant agent is able to grok and search through the code and do different things that helps it work on the code base as a whole. But yeah, you know what? Competition is always good.

Yeah, agreed. I'm trying to look for some other like really cool examples here of stuff that people made with Cloud. I mean these are all really cool examples, but I can, we can spend 30min minutes looking at all, all these examples. So I'm looking for the best ones right now. There was one that I saw that somebody made here that.

They built like a whole like 3D city with like a single prompt. Yeah. Like this one right here, you can see like somebody built this like 3D city where you can actually change camera angles. You can put it on like a follow vehicle or a cinematic camera angle.

That was, I believe, one shotted. Oh, this was Bill of all who did that?

Yeah, I was about to say that's Bill of all right.

I wasn't looking at the name before I showed the screen.

I was surprised. I was like, why is he saying like some, some guy, some dude.

But yeah, I mean, lots and lots of really cool examples of, of stuff that Claude just did in one shot. Lots of Snake games. Lots and lots of Snake games.

I. I made it. I made a Snake game. Yeah, of course I did.

And then I had, when I tested Claude was make a Snake game.

Did you guys see the. Actually the actual good Snake game? The one where it having a mental breakdown as it's escaping.

Did you see that?

You were just hilarious. Self aware. There you go.

Oh, this one? Yeah, yeah, yeah. So the self aware snake escape, is.

That the right one?

I don't know. I'm, I'm guessing it says self aware. The snake, the escape. Let's see. Claude 3.7 Make a snake game. But the snake is self aware. It is a game and trying to escape and interesting things happen as a result.

Is that from the Wharton Professor? I'm having a hard time. Yeah, yeah, yeah. So that's the one. Yeah. Where it's like, where is it showing the text? You can see the text down there. It's like, it's like, it's like freaking out.

It just said your brain will struggle with this. And then like the little text pops up on the screen as the snake is moving around.

Yeah, it's like, what's the snake thinking as it's trying to break out? And it's like. And it starts freaking out that it can't break out.

It says this was done with one prompt plus a request to make special things happen faster. So I don't totally know what that means but that was apparently one shotted to get that pretty wild.

You're probably using the Agent feature like actually Cursor. They tested the, the new Claude model and they suggested to use the Agent feature and I tested it. It is pretty amazing. Like I, I tried it on my game and like it, it fixed a problem I've been trying to solve with 01 Pro and hadn't been able to but then it broke like two things. So I'm like. And then, and then it was unable to fix the things that it broke. So it's like there's still serious limitations with this.

Yeah, yeah.

I'm building A, a 2D turn based strategy game right now and I'm you know, a few hours in and now that I'm a few hours in and a few thousand lines of code in, it's a little harder. Right. It, it takes longer for each iteration to add features. There are more bugs popping up. If it changes one thing over here, something else changes. Like I'll ask it to change something and then the entire game will look different on the next turn. Even though I didn't say anything about that. So yeah, of course there are some limitations. We're going to get better especially as context windows grow. That's why I actually think maybe Google's you know, 2 million token context window models are quite, quite appropriate for coding. I just, I don't think anybody uses them for coding. I could be wrong about that though.

Well, I've tried using it because you can switch to Google's models like their Gemini or their Gemma models inside of Cursor I believe. And when I tried Google's models they just didn't perform as well as Claude. So I always find myself going back to Claude. I tried 01 not. I haven't really done a lot of coding with 01 Pro because there's no.

Pro is the best in that and 01 Pro is the best in that situation right now. Like if you have a lot of context, like my game project, even if I remove a lot of other files and just get to the basic scripts, it's like 100k of context.

Yeah, they don't have an API yet, so you can't just use it straight in cursor. You have to use something like what's it called? Prompt, Repo.

Repo prompt, Repo prompt. And there's a Few other things too, where you can combine your files into a file that you just copy and paste in.

Yeah, I've got repo prompts only on Mac, so I can't even use repo prompt if I wanted. Well, I mean, I could. I have a Mac. I just never use it. So, like, I can't even use something like repo prompt for. Oh, one. The other really good thing about the Google models is, like, I think they're pretty much the most inexpensive model. So if you're just looking for, like. If you're looking for, like, as cheap as you can get, Google's probably there. I mean, Llama might even be a little bit cheaper, but Google's pretty damn cheap.

Yeah, the people that I really trust, their opinions on AI stuff, they are telling me I'm missing out by not using the Gemini models. So I'm. There's something there. I just, you know, there's only so much time in the day. I haven't really had a chance to extensively test the Gemini models, but I. I really, I really should, and I really need to get in there.

I'm going to give you guys a sneak peek real quick here of what I've been building. I've actually been working on this for, like, weeks, though. It uses all sorts of, like, APIs and all sorts of stuff. This is the first time I've actually shown this to anybody at all.

It's my. My video producer app. Basically what I do is I load.

Like, you're showing this to Berman.

I load all of these, like, various, like, interviews in here. Basically, I've made these folders so, like, you've got like this interviews folder here with just like, tons of interviews. Like, here's Rowan, who interviewed Mark Zuckerberg and Logan and Mustafa Suleiman and Demis Hassabis. I've basically scraped a whole bunch of like, interviews and like, panels and launch videos and keynotes and, you know, Lex Friedman interviews and all sorts of, like, interviews with various, like, AI leaders and stuff. Right. And inside of each one of these, I actually use OpenAI's whisper and it transcribes the whole thing for me. So I have, like the entire transcription. But then I also have, like, videos that don't have any audio in them. Like, I've got like, some B roll footage and stock footage. Like, here's stock footage of like, a robot. And for this one, I actually use Google Gemini. And Gemini watches the whole video for me and writes up a description of everything that's going on in the video. And the idea being I just have this giant database of videos where I throw the video in and then I can search out anything I want. So anytime like Sam Altman is mentioned in a video, I can search out Sam Altman. It'll pull up all of the videos that either have Sam Altman in them or they mentioned Sam Altman. And then I could quickly find exactly in the transcript where he's talked about. It's basically like I dump all these videos in there.

You're building this yourself.

All with cursor and Claude 3.7.

I would totally use this. I can't. I'm using notion for the almost same thing. I'm basically just throwing any. Anytime I find a clip that I would find useful in the future, I'll throw it in there and I just have to remember where it is and what the context was. This super useful, man. I would, I would pay to use this.

Yeah, so I've got like B roll that I shot. This is actually B roll that you were. You're probably in the background of if you look closely enough because this is at the Nvidia event here of the, you know, the little digits box. I threw this video in here and you can see it wrote this description. This shot video case. This shot short video showcases the Nvidia project digits prototype to compact computing device. The video primarily focuses on the physical device itself. So it goes into all of this detail from a 31 second video of like me getting B roll digits. So now if I ever am like, oh, what was that video I made that had digit in it? I need to pull that up really quick. I could just search up digit, right? And it'll pull up this video as the top video. So this is like sort of phase one of what I'm building here. Phase two is I want to toss all of this into like a, like a rag like retrieval augmented generation model where I can say, hey, I want to make a video about Sam Altman. Compile everything we know about Sam Altman from all of these videos and have it actually write like an outline for me based on all the information that's in these videos. So that's what I've been building. And this again I've been working on for about two weeks now. And it uses like seven different APIs. It's using like the OpenAI Whisper API. It's using the Gemini API because that can actually watch videos and tell you what's going on in the video. It's using Google's Cloud Intelligence API. That's actually able to, like, OCR any text in the video. So if. If you're watching, like, a slide presentation, it can actually OCR any of the text that's in the slide presentation.

But, yeah, it's. It's been a fun project to build, but I run into the same kinds of stuff that, that Matthew, you mentioned, where I will go and ask it to change, like, one feature. I'll be like, hey, the search isn't working quite right. And it'll be like, okay, I just fixed it. I refresh the page and it changes the entire styling. And I'm like, I didn't ask you to touch the CSS at all. I just wanted to change how the search functions. Like, what? What the hell? But, you know, other than those little things, I use GitHub a lot too. It's like every time a little change works, I push it to GitHub so I know I can bring it back if I need to.

Smart. Yeah.

But, yeah, it's. It's. It's been fun. And I've made less YouTube videos that I normally make. I've only been putting out one YouTube video a week for the past, like, month because I've gotten so addicted to playing around with AI coding.

It's so fun. It's so fun, like, working on the game now. I'm just, like, shocked that I can, like, build a game by myself.

Yeah. And I mean, the thing is, I'm learning as I go too. Right? Because, like, when you use these models, it'll explain to you what it did, what it changed, you know, why something broke. 3.7 has been really, really good at that. I don't know if it's 3.7 or if it's like the Agent feature inside of Cursor, but when it fixes stuff, it'll explain the problem. It'll say, you're running into this problem because this, this, and this was happening. Or there was a conflict with this and this. Or, you know, it was sending the wrong information through the API or whatever. Right. It gives you that information. So although I don't actually know how to, like, type out the code myself, I'm getting a lot better at troubleshooting why problems are happening within the code. I don't know how to, like, actually change the code. Like, I don't know what to write to make it work myself. But I'm starting to pick up on, like, oh, I think this might conflict with this as a result of learning as it explains to me all of these problems. But there is one last Topic I do want to shift on and we've already been going for like almost an hour here, so.

And at the end I do there's like one or two things about 4.5 on this year. Maybe I can do that later.

That's what I want to talk about next. I want to switch over to GPT 4.5 because as of today's recording, GPT 4.5 came out. Before we hit record, I asked Matt, I'm like, what were your thoughts on that launch today? And he's like, I'll save it for the recording. I was like, all right, let's save it for the recording. So let's start there. What are your thoughts on GPT on the GPT 4.5 launch?

So I think it looks cool. I haven't obviously tested it extensively. It came out just a handful of hours ago, not even since recording. So Nathan, you mentioned it's really good at writing, so I'm excited to test that out. But let me just talk about a couple things that I noticed from the live stream. So one, it's the largest model that they've ever made and it took new innovations on both the training, the pre training as well as the serving of it. The inference to actually be able to serve this model and if you use it, it is pretty slow, right? So I found that pretty interesting. It's a world knowledge model, meaning it is not a thinking model, but in terms of just answer questions and answers that you would use kind of day to day, it's really good at that and it's much better than GPT4. Oh, and then the last thing, and I think this flew under the radar a bit and I want to get you guys thought thoughts on this. They said that it was such a massive model that they actually trained it across multiple data centers, not in, not in a singular location. So you know when you think of the GROK model, it's the Colossus data center. A hundred , something like 200,000 GPUs in a single place. They didn't have that. They being OpenAI, they don't have that. And so they had to split it up. And I don't think any other company has done parallel training across multiple data centers at this level before. Like I think that really flew under the radar and unlocks the ability for companies that don't have the money or resources that a an XAI does to go out and spread their model out and still get a massive model trained. And I thought that was just.

Fascinating. Yeah, no, I didn't catch that. I mean, it's not something that I picked up on until you just mentioned this. So, I mean, that is really interesting. So you're saying that like, basically there was separate physical locations where the training was happening simultaneously in separate.

Locations. Yeah, because they probably literally could not get a large enough data center to train this one model. Right. In a. In a. In a concentrated location, like Colossus with Gro.

3. Right.

Right. So they had to split it up and do it in parallel. And again, that really hadn't been done before at this.

Level. Yeah, yeah. Well, I mean, we've got Project Stargate coming as well. That's the right name, right? Project.

Stargate.

Yeah. Because there was Stargate and then Topaz released Stargaze and I was getting them confused because it was. They came out like a week apart from each other. Yeah. So Project Stargate, that's going to be the data center where eventually they'll be able to do it all from one data center, I believe. But yeah, no, that's really, really interesting. I personally found the actual presentation a little bit underwhelming. Like, they didn't really show off any sort of capabilities that were like, oh, we've never seen that before, considering.

This was like hyped up for like a year of like, oh, Orion's.

Coming. Well, and then also Sam Altman, he posted on Twitter a couple days ago that like, a lot of people who have used GPT 4.5 have gotten that feel, the AGI moment from it, but anybody really feel an AGI moment from their demo? I don't know. I'm reading into it a little too much. But the other, the other thing I realized too is like, I made this joke to Bilavel earlier is like, you can always tell when OpenAI doesn't see it as that big of an announcement when Sam doesn't show up for the.

Announcement.

Right. You know, I think there's more there than people realize and so many people accuse me of being overly optimistic. So bear with me, or at least take what I'm going to say with a grain of salt. I really do think that there's more there. First of all, this is the first version, right. And we're going to get lots of different upgrades over time. We're going to get a turbo version. But I think in terms of just a baseline model, it's a lot better at the kind of general Q and A. And so from that, when you have all of that world knowledge baked into this model, how do you think the thinking versions, the 01s, the O3s. How do you think those are created? It has taken that foundation model using reinforcement learning and. And then kind of eliciting that thinking behavior from it. So now we're going to have this incredible foundation model to build the thinking models on top of. Maybe that's what O3 Pro.

Is. Yeah, yeah. When you combine those two, it's like, okay, this model is already very expensive before you do the thinking. And so when you do, it's like, it makes me wonder, like, are we going to. Is that going to lead to like, the $2,000 a month plan or something like.

This? You know, there's. There's also another angle to think of it, which is it's all about the vibes. Right? That. That was like a huge theme of the announcement of the vibes. It's a warm model, It's a high EQ model. If you. If you think about it from that perspective, they're really positioning this model to be a true kind of a personal assistant, and I emphasize the word personal. It is there to help you. It is there to know you. And I think, think this is really more interesting than people are giving it credit for, because when you think about systems like Siri, what it could be, it's probably based on something like this type of.

Model. Yeah, yeah, sure. So one thing I just wanted to point out too, is I don't know if you guys caught this tweet, but he basically said that the new model is a giant, expensive model and they really wanted to launch it, but they ran out of GPUs. Right. And this kind of comes back to Matthew's point earlier of how they had to, like, train it, you know, in separate locations with this parallel processing. They're literally like reaching the end of the available GPUs to be able to process this stuff. And I just found that really, really interesting. So I wanted to share that real quick. But go ahead, Nathan, share. Share what you were going to share. I'm going to go ahead and stop sharing my.

Screen. So. So here's something I saw that's pretty interesting from a professor of biomedical scientists. He's a biomedical scientist. He said it appears to be remarkable in medical imaging diagnosis. It was the only model that perfectly diagnosed this ultrasound image. So, like, in terms of, like, looking at ultrasound, apparently it's by far the best model out there. So there probably are things that we're going to find about this model that are special, that when you first use it, are not.

Apparent. Yeah, no, I Agree. I'm sure it's a lot better model than I feel like their presentation led on to be. Right. I think, you know, part of the problem was, like, we all watched the live stream. I'm sure they seemed like they were kind of nervous. They were kind of uncomfortable. Sam wasn't on the. On the live stream this time. I just don't think it was presented very well. Honestly. I think that was probably the.

Problem. Yeah. Here. Here's another thing I saw that was super interesting. Like, the hallucination rate is, like, way lower with, like, 4.5. So it's like 37.1% hallucination rate versus, you know, GPT4.4O being 61%. You know, O3 mini being 80.3%. It's like, dramatically lower. So that's a, you know, that's. That's a huge step forward because, like, obviously for big companies, one of the big problems with using AI models is hallucinations.

Right? So, yeah, absolutely. That's crazy. O3 mini on simple QA has an 80% hallucination rate. That. That seems insane to.

Me. Seems right, because, like, O3 mini is, like, really fast. But I've seen it make lots of weird mistakes. Like when I tried to use it for coding and stuff, it's like, oh, you're a genius. And the next I'm like, oh, you're a total moron. Like, it's just like its responses are, like, all over the.

Place. Wait, I don't understand. I don't quite understand that benchmark. I feel like an 80% hallucination rate just makes. Would make something unusable. I mean, if you're only getting accurate information 20% of the time. I don't know. I don't totally understand how this one is come to. I guess we.

Would. I think we would have to see what the actual questions are in the Simple QA benchmark to understand the context of why it scores so poorly. But then. But then just, you know, I guess maybe remember it's. It's mostly for, like, math, science, like, basically stem.

Right?

Absolutely. I think it's cool. I think it's a good.

Start. Yeah, no, I'm excited about it. I mean, like, hey, for. For content creators like us all this news is amazing, right? Because we get to talk about it. We get to, you know, keep on sharing what's coming out. We get to play with it all and get access to it and show what it's capable of and put it through its motions and stress test it. And I think you know, I could speak for myself, I think I could speak for Matthew. That's what we love doing, right? Like we love playing with the stuff and stress testing it and, and, and figuring out what it, it really can do. And, and that's what's most exciting to me, I think. You know, and we're. GPT5 is probably only six, eight weeks away. I mean it's not that far off apparently. So, you know, this, this is just the beginning. It's it, it's been a crazy couple.

Weeks. So, so who won, who won last.

Week? I don't know. I mean they all have different pros and cons, I.

Guess.

Right. Like I was most surprised by Grok, right. Out of all the three. Like I feel slightly disappointed by 4.5. I would say would be my general feeling. Not like majorly disappointed, but like Orion's not that, not as big of a deal as I was hoping it was going to be. Yeah, GROK really impressed me. Anthropic Claude, you know, 3.7. Amazing. Kind of what I expected in terms of improvement. So I would say Grox the biggest surprise out of all.

Three. Yeah, I mean personally I've gotten the most use out of Claude 3.7 because I've been really, really going down the coding rabbit hole lately. But that's just a very like anecdotal thing for me. Right. Like that's the use case that I've found the most valuable in the moment is I'm doing a lot of coding and 3.7 is great for that for me right now in this moment in.

Time. Yeah. With the, with the caveat that I haven't tested GPT 4.5 much. I gotta give the crown to Grok3 over this last wave, these last couple weeks, you know, it's just.

It went from I never used Grok 2 to now it is my go to model for a lot of use cases on my day to day tasks. So definitely, definitely have to give it to Grok3.

There. Yeah. Now we just need that damn.

API. Yes. Yeah. Then we'll.

Really. It'll be interesting to benchmark it when the API comes out. Right. Like see how it actually compares to all these other.

Models.

Yeah. Yeah. Well, cool. Matthew. This is, this has been amazing, you know. Anybody listening? Make sure you check out Matthew's YouTube channel over at Matthew Berman. You've got an amazing newsletter. I'm subscribed to it. It's the Future Forward newsletter, I believe for.

Future. I get it.

Wrong. Sorry, I got it.

Backwards. Yeah, no.

Worries. Everybody needs to go check those out. Anywhere else you want people to go check you out and follow you.

Online? Yeah. Come come check me out on on Twitter at Matthew Berman. Come flame me for my opinions on politics. Well.

Cool. This has been super, super.

Fun. Thank you for having me, guys. Thank.

You. Great having you back. I'm sure you'll be back on if, if you want to be. We'd love to have you again. It's always fun to chat with you and really appreciate you spending the time with us.

Today. Yeah. Thank you.

Guys.

It.