Grok 3 vs Claude 3.7 vs GPT-4.5: Which Update is The Best?

Hey, welcome back to the Next Wave podcast. I'm Matt Wolf. I'm here with Nathan Lenz. And there has been an absolute ton of AI news that came out recently, especially in the world of large language models. We got Grok 3, we got GPT4.5, we got Claude 3.7. Just so many new big foundation models have been released. And so for this episode, we wanted to deep dive into what each one is good at and what each one is not good at. So for that, we brought on our good friend Matthew Berman, who is probably the best person we know to really compare them all, because he deep dives and tests every single one of these models way deeper than we test them. So let's go ahead and just dive right in with Matthew berman.

Using only 20% of your business data is like dating someone who only texts emojis. First of all, that's annoying, and second, you're missing a lot of context, but that's how most businesses operate today, using only 20% of their data. Unless you have HubSpot, where all the emails, call logs, and chat messages turn into insights to grow your business, because all that data makes all the difference. Learn more@HubSpot.com.

Thanks so much for joining us, Matthew. It's great to have you back on the show.

Thanks for having me.

I've been telling people for a long time, I think I even mentioned this to you last time we had you on the show, that when it comes to large language models and trying to compare them and talk about which model is best at this and which model is best at that, I don't even do that on my YouTube channel anymore. I just point people to your channel. I'm like, yeah, Matthew's gonna test this and he's gonna tell you which models do what better than others. So you're like my go to now when it comes to, like, comparing large language models.

I appreciate that. That's awesome. Thanks for having me again. I had a great time last time, so I'm excited to chat again.

Yeah, yeah, likewise. So this will be fun. I'm trying to figure out where the best place to dig in is because so much has come out. You know, here's kind of the timeline of events that I feel are the important events, right? We got Grok 3, and then a few days later we got Claude Sonnet 3.7. And then a few days after that we got GPT 4.5, which actually happens to be the day that we're recording this episode is the day that GPT 4.5 came out. So we've all had that news in our heads for three hours now and that's about it. But maybe we start with Grok. Matt, what have your thoughts been on Grok 3 so far? Like how much have you played with it and what have you found it's like really good at so far.

Yeah. So if I could show you my bookmarks bar in Chrome, you will see that Grok now has a prominent placement right next to ChatGPT and right next to Perplexity. So the answer is I use it a lot and it has really become my go to large language model.

Oh really?

Yeah, same here.

Look, I have been a pretty die hard chatgpt user. There's really two reasons. Number one is speed. Right. I think speed of these models, speed of the response is really underappreciated by a lot of people. But it's the same reason why you convert at a higher rate when you know a webpage loads faster.

Right.

It just builds trust and you just get the answer more quickly. And then it's also the real time information, having access to all of the news on X in real time and it's such a killer feature and of course it just, it's a good model. So I think those two factors plus it just being a fantastic model that has made it my go to model. But that might change now that 4.5 is out. But we'll get there.

Yeah, yeah, yeah.

One thing I would say too is they say that they're really improving the model like really fast and I believe it cause like their team actually reached out to me and they're like, what's your feedback? And I've been going back and forth with them like giving them my thoughts on what they should be doing and I was like that's cool. Yeah. And sorry if, you know, if Aravind or anyone from Perplexity hear this but I'm like this has replaced Perplexity for me and I think you guys should double down on that. That's what I was telling them. I was like, I think you guys should be like replacing Google, replacing Wikipedia, replacing Perplexity. Any question I have, I should be going to Grok. You know they're combining search and X data. Like no one has that so. And no one ever will have that.

Yeah.

Nathan, does that make you a high taste tester? Oh yeah, because like I'm, I'm certainly not. I get no previews at all.

Yeah, I don't either. I don't seem to get any early access to like any of the anthropic models, the OpenAI models, the X models? None of them. Yeah, I think that Nathan probably has the best ins right now.

I was a low level member of like a lot of the different Silicon Valley mafias. You know, I was not like a high level member, but I was like the low level of like, like several of them. You know, the gaming one, the social media one, the crypto one, the Taiwanese mafia, I was kind of members of all those and some of the YC people as well. I know a lot of, a lot of them too.

So, you know, I personally think that Grok's biggest hurdle is just Elon. I'm sure you've seen this, Matthew, on your YouTube channel because you've made a couple videos about Grok. Now when Grok came out, I made a news video about it and overwhelmingly all of the comments about Grok are like, I'm never going to touch anything Elon makes. You know, and so I almost think that like so many people are throwing the baby out with the bathwater when it comes to actually using and trying Grok just because it's Elon's name attached to it.

Yeah. So whenever I post anything about Grok on X, if it's negative about Grok or Elon, I get flame.

Right.

If it's positive, everybody cheers it, shares it, everything. The opposite is true. On YouTube, I made a video about Grok3 being really good because it is. So many people commented they won't touch it, just like you won't touch it. Elon's, you know, AI model bias, conservative, right wing. I'm apolitical, you know, I'm trying to stay out of it as much as I can in the videos, but anytime I mention anything having to do with Elon X Grok, I get the meanest comments on YouTube.

It's wild. I just, I think so many people are just going to like, not experience probably what is the best model available right now just because of the Elon factor. Now when it comes to code, I don't necessarily think GROK is the best. I don't know. I actually honestly haven't tested Grok with code because I mostly do code with cursor. And I don't believe Grok3 has pushed its API out yet. I don't think, I don't think so. So I've been mostly playing with 3.7.

I've tested it. Yeah, I've tested a little bit. I mean, you know, it's weird, it's not very consistent, like there's sometimes where it's amazing at code and you're like, oh wow, that's like a really creative solution and it's doing something better than Claude in like some very limited circumstances. But then other times it just won't follow my instructions as well for code in terms of reliability, like when Claude 3.7 came out, that's by far the best now for coding. But it feels like there is something there. Like, you know, I wouldn't like, no, discount them forever. Like they could improve that and all of a sudden Grok 3 could be the best at coding.

Right? Matthew, I'm curious. So when it comes to Grok 3, I know you have kind of like your own like internal benchmarks that you've been using on some of these tools. Like what have you found like Grok3 is really good at? And like are there any things Grok3 is just like not going to be your go to for?

Well, let me say something first I had to throw my benchmarks out because they were completely saturated. They were absolutely annihilated by every single bottle that comes out nowadays. So I threw em out. I'm currently in the process of creating a new set of benchmarks and questions GROK again, like the thing I go to it for is real time information as quickly as I need it and that it's awesome at. I've tested it on some other things like quick coding challenges, some math challenges, and it does really well, but those aren't the everyday use cases for me. So there's only so much I can test with it. I'll just prompt it with one of my benchmark questions, see if it's right and say okay, yeah, it's right or no, it's wrong. But overall what I care about, and I think what most people care about is day to day usage. Is it going to solve my problems? And it does. Grok3 is great at that. And you know, typically I was using GPT4O, now Grok3 with thinking kind of replaces that. Although I don't really need the thinking. Yeah, it's just really good at the day to day stuff that I would use it for. So I'm kind of split between, you know, Grok3 perplexity and ChatGPT.

And now 4.5 came out. I would say I tested it right before we got on here and I would say okay, 4.5 has a better vibe for general chat, but it's slower.

Oh, it's about the vibes.

But when you get the response back, I Do like the responses better, actually. So that's impressive. Like, it's writing is better, but it's slower. So it's like there's a huge trade up there. It feels like 4.5 is going to serve some, like, maybe for like creative writing or creative work. It's probably the best for now. But it is interesting that the cost is so much higher.

Really?

Yeah. I think it was 70 per million input tokens, half of that for cashed input. It's funny, Nathan, that you bring up writing that is honestly something that I want to use AI for much more often than I do. But it is so bad. It's so bad at writing. And I haven't had enough time to test 4, 5 for writing. But I really hope that it's good. It's better because then that's going to be able to help me with a lot of things that I do day to day. I don't script my videos, but sometimes I want help writing the bullet points for them. Or sometimes I want a, you know, a tweet thread. The initial drafts created for me, I.

Gave it probably, I don't know, over a hundred pages of different notes about my game, including the story and things like that and the game mechanics. And it took a long time to respond. Actually, like a very long. I was like, kind of surprised. Like, this feels like 01 Pro when you hint into a lot of stuff. 4.5. 4.5 took a long time to respond. I was surprised, like, very long.

It's very slow. Yeah.

Took like three minutes to respond or something to all that. But then its notes on the story were, like, perfect. It gave me, like, amazing critiques of like, I love this part. And it even like, had like little emojis having color codings. It had like green, orange, blue and red and different, like, ones like, you know, it was like, green's good. Love this part. This part probably could be tweaked. This part. Don't like it. Here's why. And these parts are interesting. Maybe you keep them, maybe you don't. The feedback was good. Like, it was solid. So I do want to test it more for writing, but my first impression is, yeah, it is probably the best model for writing now.

That's great.

Yeah. Well, going back to Grok for a second, have you guys played with the voice mode yet?

I've used advanced voice mode a little bit. I don't find it to be, like, super useful throughout my day. If I'm driving, if I'm walking, if there's just not A screen in front of me, maybe I'll use it. But what I found specifically with ChatGPT advanced voice mode is you'll ask a question and there's that delay, and then sometimes it repeats it and then stops and then repeats it again. And it's just such a high friction experience. It's not great.

Yet I do use voice on Perplexity. I'm a big baseball fan. Spring training just started with baseball. They've introduced some, like, new rules that I didn't realize existed. I was watching one of the spring training games. I opened up Perplexity. I'm like, hey, is there a new rule that I didn't know about? And I just will talk to Perplexity and ask it questions, and it will, like, do all the research, figure out the new rules that are going on in baseball, and then give them back to me. And I found that pretty helpful. I still find myself typing my questions more often than speaking them, but every once in a while I'll be feeling lazy. I'll hit the voice button and just ask my question. I find that helpful. I know. Nathan, you kind of almost use some of the voice modes, especially advanced voice and chatgpt, for, like, journaling, Right?

I'm a big believer in it, you know, long term. But I have been kind of disappointed. Like, an advanced voice mode, like, the demo they showed seemed amazing, and it seemed like they removed so many different parts of it. And some of the things they demoed when you tried them in real life, they don't work as well as in the demos. My wife's Japanese. My Japanese is getting better, so I don't have to use it as much. But for, like, really hard topics, we have tried to use it to do translation, and it just gets confused so easily. Like, as soon as you go from one language to another, like, sometimes it'll translate it properly. It's like, okay, cool. It worked in that use case. But okay, well, then she talks back to me. Then sometimes it just totally gets confused. And instead of translating, sometimes it'll, like, start talking versus translating, start saying its own stuff versus doing a translation. And it's like, okay, that's super annoying. And we just turn it off every single time that's ever happened.

Yeah, I know. Sam Altman's even talked about. He wants the AIs in the future to be more in line with your own beliefs. Right. So, like, it knows your religious beliefs, your political beliefs, to bring your own bias. Yeah. Your own bias. It will actually learn your bias and sort of lean into your bias more to give you more of what you want. And I think that's a really, really scary thing. There was a little bit of word about this happening on, like, Facebook, right, where Facebook was using, like, AI bots, the feed that people didn't even realize were AI bots. And so people would post stuff on Facebook, they would get, like, a bunch of responses from these AI bots, not even realize that they're AI and they're going, oh, cool, I get great engagement on Facebook. And it keeps on bringing them back to Facebook over and over and over again, because Facebook's where they get engagement and they're not even realizing that they're talking to AIs. I think that's going to be a bigger and bigger problem as well, where, you know, social media is all about getting dopamine hits, right? We post our tweets because we want to get those likes, we want to get those retweets, we want to get those comments. Every time we see one of those, we get a little dopamine hit, and we keep on coming back for more. Well, if AI gets really, really, really good at giving us those dopamine hits every time we want them, we're going to go to wherever we get the most dopamine hits, you know, at the highest frequency. I think that's what's really sort of worries me about the future, and it ties into, like, the whole population collapse as well. I think it gets to that point where people communicate with other humans less and less and less and less because they're getting their dopamine hits from fake people on social media. They're getting their conversational needs met by unhinged voice chats that, you know, have the same bias as me. They love Trump and they're, you know, the maga girl. And I can talk to the maga girl who has the same belief system that I have or whatever. I'm not saying that's my belief system. I'm just, you know, for example. And so it's very, very concerning. Like, I think you and me, Matthew, are pretty much on the same page, where we generally lean optimistic on this stuff. But there's still quite a few things that actually do scare me about this as well. I'm not like the whole accelerationist where I'm just like, push forward as fast as possible. I'm like, maybe there's some things we shouldn't push forward as fast as possible on. And that's definitely one of those areas.

Yeah.

You know, Elon had said with Grok which I find kind of promising hopefully, is that, you know, they want to be maximum truth seeking, which sounds a lot better to me than what I'm hearing from OpenAI of like, bring your own bias. Like we're going to allow you to pick your bias and then we're just going to serve you up information based on that because all truth is subjective or whatever. But what I worry about, you know, is obviously Elon has his own bias now as well, you know. And you know, I'm probably more right wing than anyone on this podcast right now. I'm not super right wing, but you know, after living in San Francisco for a long time, that kind of did that to me. And I do worry that he'll go too far with it and GROK will become like a right wing AI. I do think it needs to be unbiased. I'm not sure how you do that because there's bias in everything. Like if you pull up facts online or you pull up something from the Wall Street Journal or CNN or whatever, there's bias in all of this. And so since all the models are trained on that, I'm just not sure how you get around the bias.

You know, by the way, this is the part where I start to get flamed on Twitter and YouTube, but for completely opposite reasons. But I'm still going to share anyways. This is not reflective of my political opinion or anything like that. But here's a couple things. Yeah, Elon has said maximally truth seeking for Grok 3 and great, like in theory that makes a lot of sense, but how do you do that? Ultimately you have to create systems and maybe you're able to create them or maybe you're not, but those systems are created by humans. The original training data is created by human by humans. The post training techniques are created by humans. The reinforcement learning is also set up by humans. So there's like a human in the loop. So what he claims to be completely maximally truth seeking is just what he believes. And maybe not necessarily him, but he's trying to give a different perspective, which, you know, fine, but did you see what happened just a few days ago? Grok3 was caught having custom instructions that said don't cite any sources that say Elon Musk and Donald Trump are spreaders of disinformation and they were caught doing this. And so I reported on it and of course flame from both sides, but fine. But the point is, yeah, it's as simple as somebody submits a PR and it has a little line in the system prompt saying, don't do this, and then all of a sudden, fingers on the scale. Right. So it's very possible. It's not only possible, it's happening. And it doesn't matter if it's from OpenAI or Grok or Google or anthropic. They all have bias. It's. It's. I can't imagine a world in which the bias is removed completely, although I hope it does. I just, It's. I don't see a path towards that.

Yeah, I mean, I honestly don't understand how it's possible. Like you guys said, it's all trained on data that was created by humans. It's all basically a scraping of the entire Internet, which just inherently has bias. Like, I just, I don't understand how it's possible, honestly. And how do you determine what is the truth and what is not?

That's the question.

I don't know. Maybe if the models get really smart and can do, like, real reasoning, and if they can look at both sides of anything, they can kind of find the middle ground where there's some truth, you know, where there's actual truth, hopefully. But then some people will perceive that real truth is not true. It's like. Well, people will perceive it to have bias, even if it ends up not having bias.

But I feel like you're getting into, like, this world where now we're letting AI decide what is and what isn't, like, ethical. Right. Like, now we're letting, like, AI sort of decide what is, like, philosophically correct and what is not. And to me, that seems weird to let machines do that for humans, you know?

Yeah. And, you know, it's funny you say that. I mentioned the same thing. And Dave Shapiro, another fellow YouTuber talking about AI, said, you know, actually, I would rather just completely give it over to AI to make ethical decisions. I think that's what he was saying. But it's interesting. I can see both sides of it. If there was a system that was completely unbiased, assuming. Yeah, okay, let them decide. But how do you make that?

As you said, I think you can't allow AI to make that decision. If you allow AI to make that decision. I mean, that goes towards eugenics and crazy things like that. You can't allow AI to optimize based on human performance or some crazy metric like that. That would just lead to horrible things. I think I. Yeah, I totally disagree with that.

Yeah. Yeah. Let's shift over to Claude, because Claude 3.7 came out a couple weeks ago and that one has proven to be really, really good at code, it seems. Right. It seems like it didn't make huge improvements in almost any other areas, but it got a lot better at code from what I've seen so far. I mean, Matt, you might have some different experiences with it. I'm using Cursor to write code and it's definitely gotten better at coding for me. I've definitely noticed things that it would run into these like loops where it wouldn't write the code, wouldn't write the code, couldn't figure out what the problem was. 3.7. One prompt was 3.7. Now it figures out a problem that I had gone back and forth 10 times prior. So to me it seems like it got a lot better at code. But from most other people's experiences, it seems like it didn't really improve in almost any other areas other than code.

Yeah, when I think about coding, I think about Claude. Right. Claude 3.5 was the go to model for a lot of coders using AI assistants to help them code. Yeah. Cursor + Claude, windsurf + Claude, that was the model. Now we had this huge upgrade and I agree, it is a huge upgrade. It also has the thinking capability. So I've been doing stuff vibe coding, right. So I'm using some kind of ide. Whether cursor or windsurf plus, I'm using 3.7 thinking. And it's fantastic, right? It is fantastic. Now I'll bring it back to what I said earlier. The real world use cases, the stuff that I want to use it for day to day one is coding, but a lot of other things are not coding. And here's the thing, it's not that fast and it doesn't have web search, it has no real time information. So it's essentially unusable for me outside of coding.

Right.

Although it's fantastic at coding, I'll give you that.

Well, the nice thing is if you are using something like cursor, Cursor can actually do the web search for you and then give that additional context. So it's almost like cursor will actually kind of do like the perplexity thing, Right. Where if it needs to figure something out, cursor itself goes and does the search and then provides that information to Claude. At least when I've been using it. It seems to actually search the web when using cursor.

Yeah. And Claude 3.7 sonnet is available in Perplexity. I'll just mention quickly, by the way, Perplexity just adds the Latest models, all the for free and I have not been paid by them at all. I'm just a huge fan of their product.

Yeah.

So, yeah. So if you wanted to try any of these new models, you could go try it if you already have a Perplexity account.

Yeah. And their deep research is really good. If we're, you know, talking about good things about Perplexity, if we're praising Perplexity, they just threw Deep research in there as well. And it's really good.

Yeah. But yeah, web search is critical.

I want to say about, about Claude Sonet. Like, my feeling is, you know, before it was released, Claude was my go to for just like general chat for like discussing anything. Right. Like even like my game design document. I would, I would share that with Claude and that was my favorite. Then Grok replaced that. But Claude was pretty good at code. Like you said, most engineers were using Claude Sonnet, not all. Like I was using 01 Pro and actually was better, but way slower. Right. And you had to give it tons of context. You had to do something like repo prompt or something else. But it seems like in this update they really doubled down on being the best at coding because in terms of like general chat, I feel like it actually stepped backwards, like when I use it now to chat with them, like it actually got worse in this release, like that. I don't like its responses as much. They seem less human, like, but it's way better at coding. So it feels like we're starting to see that all the AI models are finding their own specialties. Or at least that's what Anthropic's doing now with Claude. And I do kind of wonder if all the models are going to have to do that. When I talked to the people at xai, I told them double down on having the best data because you got the real time data with X, you got it with the search, double down on that. And I think you'll probably see that where like these different models will be the best at a thing. But it seems like ChatGPT is still trying to go more broad. They're still trying to be like the best overall model. And I'm kind of curious to see how all that ends up playing out.

Yeah, it's interesting, you know, quickly. So I just pulled up the Cloud 3.7 Sonnet blog post, the announcement blog post, and everybody knows Claude is great at coding and they made this huge jump in coding. But if you read it, it says in developing our reasoning models, we've optimized somewhat less for math and computer science competition problems, and instead shifted focus toward real world tasks that better reflect how businesses actually use LLMs. You know, now that I'm reading that maybe they meant like these kind of benchmark computer science problems versus real world coding problems, but it just sounds like, hey, we're not focused as much on math and coding anymore, more real world stuff. But it still is fantastic at that.

Yeah.

Did you guys see that? They're also. It seems to be that they're going to be competing with Cursor.

Oh, yeah. Because they released that code feature, right?

Yeah, Claude code. Oh, yeah, yeah. All the top engineers I know, they've tried it. It's very expensive to use, but my understanding is like, in some ways it's better than Cursor. So, like, I'm not sure exactly how it works, but I think you use it in the terminal, you get full access to your entire code base and then it can just change stuff for you.

Yeah, I tried it out. It's pretty cool.

So that is interesting. I think they really are doubling down on code. Yeah, I believe when they mentioned the benchmark stuff, I think they were probably talking about more real world coding versus, like.

I think you're right.

Benchmark coding, you know?

Yeah. I think they know that most people are using Claude mostly for coding.

So it kind of sucks for Cursor because, like, right now everyone uses Cursor to use Claude, and now Anthropic's basically going to try to kill them.

I don't know. I think people will probably still use Cursor a lot because Cursor and Windsurf, I believe, are both forks of Visual Studio code.

Yeah, they must be saying, like, oh, most of our usage is in cursor. Like, why would we not, like, just be cursor then?

Yeah, but I don't know. Visual Studio code is pretty universal. People use it for coding a lot and it's what's familiar to people. So I think trying to get people to switch to a new IDE might be a tough ask. Unless Claude themselves goes and makes their own fork of Visual Studio code. So unless CLAUDE goes and makes their own fork of Visual Studio code, I have a hard time seeing people switch over to a new IDE that's completely different.

It's also not an ide. It's literally just sitting in the terminal. So, you know, some people prefer that I prefer seeing the code more visually and have like a nice interface to deal with. So I've tried both Cursor and Windsurf and they're both great.

Yeah. Yeah. To me, they feel very same. Ish. Like I have a hard time saying which one's better. They feel very, very similar to me.

I've had more success with Windsurf when trying to iterate on a whole code base. Now, Cursor, I think, just released or kind of just updated their Agent feature, which makes it a little bit easier to operate on the whole code base. Your AI coding assistant agent is able to grok and search through the code and do different things that helps it work on the code base as a whole. But yeah, you know what? Competition is always good.

Yeah, agreed. I'm trying to look for some other, like, really cool examples here of stuff that people made with Cloud. I mean, these are all really cool examples. We can spend 30 minutes looking at all these examples, looking for the best ones right now. I mean, lots and lots of really cool examples of stuff that Claude just did in one shot. Lots of Snake games. Lots and lots of Snake games.

I made a Snake game. Yeah, I. Of course I did. And then I had.

First thing I did when I tested Claude was make a Snake game. I still always do.

Yeah.

Did you guys see the actually, the actually good Snake game? The one where it's like having a mental breakdown as it's escaping? Did you see that?

You were just showing that. Yeah, self aware. There you go.

This one. Yeah, yeah, yeah. So the self aware Snake escape.

Yes, that's the one. Yeah. It's like freaking out.

It just said your brain will struggle with this. And then like the. Yeah, little text pops up on the screen as the Snake is moving around.

Yeah. It's like, what's the Snake thinking as it's trying to break out? It's like. And it starts freaking out that it can't break out.

It says this was done with one prompt plus a request to make special things happen faster. So I don't totally know what that means, but that was apparently one shotted to get that. Pretty wild.

Yeah. Probably using the Agent feature. Like, actually, Cursor, they tested the. The new Claude model and they suggested to use the Agent feature. And I tested it. It is pretty amazing. Like, I. I tried it on my game and like, it. It fixed a problem I've been trying to solve with 01 Pro and hadn't been able to. But then it broke like two things. I'm like. And then. And then it was unable to fix the things that it broke. So it's like, there's still serious limitations with this.

Yeah, yeah, yeah.

I'm I'm building a, a 2D turn based strategy game right now and I'm, you know, a few hours in and now that I'm a few hours in and a few thousand lines of code in, it's a little harder. Right. It takes longer for each iteration to add features. There are more bugs popping up. If it changes one thing over here, something else changes like I'll, I'll ask it to change something and then the entire game will look different on the next turn. Even though I didn't say anything about that. So yeah, of course there are some limitations. We're going to get better, especially as context windows grow. That's why I actually think maybe Google's, you know, 2 million token context window models are quite appropriate for coding. I just, I don't think anybody uses them for coding. I could be wrong about that though.

Well, I've tried using it because you can switch to Google's models like their Gemini or their Gemma models inside of Cursor I believe. And when I tried Google's models, they just didn't perform as well as Claude. So I always find myself going back to Claude. I tried 01. I haven't really done a lot of coding with 01 Pro because there's no API.

So 01 Pro is the best in that situation right now. Like if you have a lot of context, like my game project, even if I remove a lot of other files and just get to the basic scripts, it's like 100k of context.

Yeah, but they don't have an API yet, so you can't just use it straight in cursor. You have to use something like, what's it called? Repo prompt.

Repo prompt. And there's a few other things too where you can combine your files into a file that you just copy and paste it.

Yeah. And I mean I've got a PC repo prompts only on Mac, so I can't even use repo prompt if I wanted. Well, I mean I could. I have a Mac, I just never use it. So like I can't even use something like repo prompt for 01. The other really good thing about the Google models is like I think they're pretty much the most inexpensive model. So if you're looking for like as cheap as you can get, Google's probably there. I mean Llama might even be a little bit cheaper, but Google's pretty damn cheap.

Yeah, the people that I really trust, their opinions on AI stuff, they are telling me I'm missing out by not using the Gemini models. So I'm. There's something there. I just, you know, there's only so much time in the day. I haven't really had a chance to extensively test the Gemini models, but I really. I really should, and I really need to get in there.

I'm going to give you guys a sneak peek real quick here of what I've been building. I'm building my video producer app. Basically what I do is I load all of these, like, various, like, interviews in here. I've made these folders. So, like, you've got like this interviews folder here with just like, tons of interviews. Like, here's Rowan, who interviewed Mark Zuckerberg and Logan and Mustafa Suleiman and Demis Hassabis. I've basically scraped a whole bunch of, like, interviews and panels and launch videos and keynotes and, you know, Lex Friedman interviews and all sorts of, like, interviews with various, like, AI leaders and stuff, right? And inside of each one of these, I actually use OpenAI's whisper and it transcribes the whole thing for me. So I have, like, the entire transcription. But then I also have, like, videos that don't have any audio in them. Like, I've got, like, some B roll footage and stock footage. Like, here's stock footage of like, a robot. And for this one, I actually use Google Gemini. And Gemini watches the whole video for me and writes up a description of everything that's going on in the video. And the idea being I just have this giant database of videos where I throw the video in and then I can search out anything I want. So anytime, like, Sam Altman is mentioned in a video, I can search out Sam Altman. It'll pull up all of the videos that either have Sam Altman in them or they mention Sam Altman. And then I could quickly find exactly in the transcript where he's talked about.

You'Re building this yourself.

All with cursor and Claude 3.7.

Dude, I would totally use this. I'm using Notion for the almost same thing. I'm basically just anytime I find a clip that I would find useful in the future, I'll throw it in there. And I just have to remember where it is and what the context was. This is super useful, man. I would pay to use this.

Yeah, so I've got like, B roll that I shot. This is actually B roll that you're probably in the background of if you look closely enough. Because this is at the Nvidia event here, the, you know, the little digits box. I threw this video in here and you can see it wrote this description. This short video showcases the Nvidia project digits prototype to compact computing device. The video primarily focuses on the physical device itself. So it goes into all of this detail from a 31 second video of like me getting B roll digits. So now if I ever am like, oh, what was that video I made that had digit in and I need to pull that up really quick, I could just search up digit, right? And it'll pull up this video as the top video. So this is like sort of phase one of what I'm building here. Phase two is I want to toss all of this into like a rag like retrieval augmented generation model where I can say, hey, I want to make a video about Sam Altman. Compile everything we know about Sam Altman from all of these videos and have it actually write like an outline for me based on all the information that's in these videos. So that's what I've been building. And this again I've been working on for about two weeks now. And it uses like seven different APIs. It's using like the OpenAI Whisper API. It's using the Gemini API because that can actually watch videos and tell you what's going on in the video. It's using Google's cloud Intelligence API that's actually able to like OCR any text in the video. So if you're watching like a slide presentation, it can actually OCR any of the text that's in the slide presentation. But yeah, it's been a fun project to build, but I run into the same kinds of stuff that Matthew, you mentioned where I will go and ask it to change like one feature. I'll be like, hey, the search isn't working quite right. And it'll be like, okay, I just fixed it. I refreshed the page and it changes the entire styling. And I'm like, I didn't ask you to touch the CSS at all. I just wanted to change how the search functions. Like, what? What the hell? But you know, other than those little things, I use GitHub a lot too. It's like every time a little change works, I push it to GitHub so I know I can bring it back if I need to.

Oh, that's smart.

Yeah. But yeah, it's been fun. And I've made less YouTube videos than I normally make. I've only been putting out one YouTube video a week for the past like month because I've gotten so addicted to playing around with AI coding.

It's so fun, like working on the game now I'm just, like, shocked that I can, like, build a game by myself. It's like, you know, in the past, you could. I never could imagine, like, one person could build a game. Or now you're building your own software product and you're kind of like a no code guy right now. You're using. You're using. You went from no code to using AI to code, and you're actually able to build a whole product. I mean, it's just everything's changing.

Yeah. And I mean, the thing is, I'm learning as I go too. Right. Because, like, when you use these models, it'll explain to you what it did, what it changed, you know, why something broke. 3.7 has been really, really good at that. I don't know if it's 3.7 or if it's like the Agent feature inside of Cursor, but when it fixes stuff, it'll explain the problem. It'll say, you're running into this problem because this, this, and this was happening. Or there was a conflict with this and this. Or, you know, it was sending the wrong information through the API or whatever. Right. It gives you that information. So although I don't actually know how to, like, type out the code myself, I'm getting a lot better at troubleshooting why problems are happening within the code. I don't know how to, like, actually change the code. Like, I don't know what to write to make it work myself. But I'm starting to pick up on, like, oh, I think this might conflict with this as a result of learning, as it explains to me all of these problems. So, anyway, that's what I've been working on. But there is one last topic I do want to shift in.

And so let's talk about it.

I want to shift over to GPT 4.5, because as of today's recording, GPT 4.5 came out. Before we hit record, I asked Matt. I'm like, what were your thoughts on that launch today? And he's like, I'll save it for the recording. I was like, all right, let's save.

It for the recording.

So let's start there. What are your thoughts on the GPT 4.5 launch?

So I think it looks cool. I haven't, obviously, tested it extensively. It came out just a handful of hours ago, not even since recording. So, Nathan, you mentioned it's really good at writing, so I'm excited to test that out. But let me just talk about a couple things that I noticed from the live stream. So one it's the largest model that they've ever made and it took new innovations on both the pre training as well as the serving of it. The inference to actually be able to serve this model and if you use it it is pretty slow. Right. So I found that pretty interesting. It's a world knowledge model, meaning it is not a thinking model but in terms of just questions and answers that you would use kind of day to day it's really good at that and it's much better than GPT4O and then the last thing and I think this flew under the radar a bit and I want to get your guys thoughts on this. They said that it was such a massive model that they actually trained it across multiple data centers, not in a singular location. So you know when you think of the Grok model, it's the Colossus data center. A hundred , something like 200,000 GPUs in a single place. They didn't have that. They being OpenAI, they don't have that. And so they had to split it up. And I don't think any other company has done parallel training across multiple data centers at this level before. I think that really flew under the radar and unlocks the ability for companies that don't have the money or resources that an X AI does to go out and spread their model out and still get a massive model trained. And I thought that was just.

Fascinating. Yeah, I didn't catch that. I mean it's not something that I picked up on until you just mentioned it. So I mean that, that is really interesting. So you're saying that like basically there was separate physical locations where the training was happening simultaneously in separate.

Locations. Yeah, because they probably literally could not get a large enough data center to train this one model in a concentrated location like Colossus with Grok.

3. Right.

Right. So they had to split it up and do it in parallel and again that really hadn't been done before at this.

Level. Yeah, well I mean we've got Project Stargate coming as well. That's going to be the data center where eventually they'll be able to do it all from one data center I believe. But yeah, no, that's really, really interesting. I personally found the actual presentation a little bit underwhelming. Like they didn't really show off any sort of capabilities that were like oh, we've never seen that.

Before. Considering this was like hyped up for like a year of like oh, Orion's.

Coming. Well and then also Sam Altman he posted on Twitter a couple days ago that a lot of people who have used GPT 4.5 have gotten that feel, the AGI moment, from it. Right. But anybody really feel an AGI moment from their demo? I don't know. I'm reading into it a little too much. But the other thing I realized too is like, you can always tell when OpenAI doesn't see it as that big of an announcement when Sam doesn't show up for the.

Announcement.

Right. You know, I think there's more there than people realize. And so many people accuse me of being overly optimistic. So bear with me, or at least take what I'm going to say with a grain of salt. I really do think that there's more there. First of all, this is the first version, right. And we're going to get lots of different upgrades over time. We're going to get a turbo version. But I think in terms of just a baseline model, it's a lot better at the kind of general Q and A. And so from that, when you have all of that world knowledge baked into this model, how do you think the thinking versions, the O1s, the O3s, how do you think those are the created? It is taking that foundation model using reinforcement learning, and then kind of eliciting that thinking behavior from it. So now we're going to have this incredible foundation model to build the thinking models on top of. Maybe that's what O3 Pro.

Is. And it'll be crazy.

Expensive. Oh, that's another thing, is the cost.

Yeah. Combine those two. It's like, okay, this model is already very expensive before you do the thinking. And so when you do, it's like, it makes me wonder, is that going to lead to like the $2,000 a month plan or something like.

This? You know, there's also another angle to think of it, which is it's all about the vibes. Right. That was like a huge theme of the announcement of the Vibes. It's a war model. It's a high EQ.

Model. They were trying to compete with.

Claude.

Right. And so, you know, because before Claude, everyone said, like, Cloud had a better vibe than ChatGPT, you.

Know. Right. So, like, if you think about it from that perspective, they're really positioning this model to be a true kind of AI personal assistant, and I emphasize the word personal. It is there to help you. It is there to know you. And I think this is really more interesting than people are giving it credit for, because when you think about systems like Siri, what it could be. It's probably based on something like this type of.

Model.

Yeah. One thing I just wanted to point out, too, is I don't know if you guys caught this tweet, but he basically said that the new model is a giant, expensive model and they really wanted to launch it, but they ran out of GPUs. Right. And this kind of comes back to Matthew's point earlier of how they had to train it, you know, in separate locations with this parallel processing. They're literally, like, reaching the end of the available GPUs to be able to process this stuff. And I just found that really, really interesting. So I wanted to share that real.

Quick. OpenAI Is GPU.

Poor? Yeah, I mean, they've got Softbank now behind.

It. Who would have.

Thought? I wanted to share two things that kind of showed that there is something special here. So here's something I saw that's pretty interesting from a professor. He's a biomedical scientist. He said it appears to be remarkable in medical imaging diagnosis. It was the only model that perfectly diagnosed this ultrasound image. So, like, in terms of, like, looking at ultrasound, apparently it's by far the best model out there. So there. There probably are things that we're going to find about this model that are special, that when you first use it, are not.

Apparent. Yeah, good.

Point. No, I agree. I'm sure it's a lot better model than I feel like their presentation led on to be. You know, part of the problem was, like, we all watched the live stream. I'm sure they seemed like they were kind of nervous, they were kind of uncomfortable. Sam wasn't on the live stream this time. I just don't think it was presented very well. Honestly. I think that was probably the.

Problem. Yeah. Here's another thing I saw that was super interesting. Like, the hallucination rate is, like, way lower with, like, 4.5. So it's like 37.1% hallucination rate versus 4.0 being 61%. You know, O3 mini being 80.3%, it's like, dramatically lower. So that's a, you know, that's step forward. Because, like, obviously for big companies, one of the big problems with using AI models is hallucinations.

Right? So, yeah, yeah.

Absolutely. That's crazy. O3 mini on simple QA has an 80% hallucination rate. That. That seems insane to.

Me. Same as riot because, like, O3 mini is, like, really fast. But I've seen it make lots of weird mistakes, like when I tried to use it for coding and stuff. It's like, oh, you're a genius. And the next I'm like, oh, you're a total moron. Like, it's just like its responses are like all over the.

Place. I don't quite understand that benchmark. I feel like an 80% hallucination rate just makes, would make something unusable. I mean, if you're only getting accurate information 20% of the time. I don't know, I don't totally understand how this one has come to. I guess I think we would.

Have to see what the actual questions are in the simple QA benchmark to understand the context of why it scored so poorly. But. But then just, you know, I guess maybe remember it's mostly for math science, like basically stem. Right. Things that have verifiable.

Answers.

Absolutely. I think it's cool. I think it's a good.

Start. Yeah, no, I'm excited about it. I mean, like, hey, for content creators like us all this news is amazing, right? Because we get to talk about it, we get to, you know, keep on sharing what's coming out. We get to play with it all and get access to it and show what it's capable of and put it through its motions and stress test it. And I think, you know, I could speak for myself, I think I could speak for Matthew. That's what we love doing, right? Like, we love playing with the stuff and stress testing it and figuring out what it really can do and, and that's what's most exciting to me, I think, you know, GPT5 is probably only six, eight weeks away. I mean, it's not that far off apparently. So this is just the beginning. It's. It's been a crazy couple weeks.

Absolutely. So who won? Who won last week? I don't.

Know. I mean, they all have different pros and cons, I guess.

Right? Like, I was most surprised by Grok. Right. Like out of all the three, like I feel slightly disappointed by 4.5, I would say would be my general feeling. Not like majorly disappointed, but like Orion's not as big of a deal as I was hoping it was going to be. Yeah, Grock really impressed me. Anthropic Claude, you know, 3.7. Amazing. Kind of what I expected in terms of improvement. So I would say Grox, the biggest surprise out of all of.

Three. Yeah, I mean, personally I've gotten the most use out of Claude 3.7 because I've been really, really going down the coding rabbit hole. But that's just a very like, anecdotal thing. For me. Right. Like that's the use case that I've found. The most valuable in the moment is I'm doing a lot of coding and 3.7 is great for that. For me, right now, in this moment in time.

Yeah. With the caveat that I haven't tested GPT 4.5 much. I gotta give the crown to Grok 3. Over this last wave, these last couple weeks, it went from I never used Grok 2 to now it is my go to model for a lot of use cases on my day to day tasks. So definitely have to give it to Grok3.

There. Yeah. Now we just need that damn.

API. Yes.

Yeah. Then we'll really be interesting to benchmark it when the API comes out. Right. Like see how it actually compares to all these other models. Yeah. Yeah. Well.

Cool. Matthew, this has been amazing, you know. Anybody listening, make sure you check out Matthew's YouTube channel. Over at Matthew Berman. You've got an amazing newsletter. I'm subscribed to it. It's the Forward Future newsletter, I believe for future. Everybody needs to go check those out. Anywhere else you want people to go check you out and follow you.

Online?

Yeah. Come check me out on. On Twitter at Matthew Berman. Come flame me for my opinions on politics even though I don't share them.

Yeah. Well, you called it Twitter. That's like, you're kind of like, all right, you're.

Right. I already offended one.

Side. I'm on your side, guys. Well.

Cool. This has been super, super.

Fun. Thank you for having me.

Guys. Thank.

You. Great having you back. I'm sure you'll be back on if. If you want to be. We'd love to have you again. It's always fun to chat with you and really appreciate you spending the time with us.

Today. Yeah. Thank you, guys.