Zucc What are we DOING?! Llama 4 Launches with... Interesting Results

I think that Meta might have officially screwed up over this past weekend. On Saturday, Meta dropped the Llama 4 series of models and we got two models from them, open source, like the previous ones. But other than that, there are some huge differences between Llama 4, their previous models, and of course the community's reaction to llama4 versus what they promised and what they showed off in their benchmarks. There's a lot to look into today from both the community and Meta, so let's dive in. Let's start off with the official Llama 4 blog blog post. Llama 4 Leading Multimodal Intelligence as you can see, they've got three models. As is typical, two models are available right now, Llama 4 Scout, which is the smaller one. 17 billion active parameters, 16 experts, 109 billion total parameters. So it's actually still quite a large model, but it's mixture of experts. So at any given moment 17 billion of the parameters are active across 16 experts. All right, got that so far. Next up, industry leading. 10 million token context length. Optimized inference, supposed to be fast while also supporting a really large context length. But the community has something else to say about that later. So that's the smaller model. Next up, llama 4 Maverick. This is the mid sized model, still only 17 billion active parameters at one moment, but this has way more experts, 128 experts for a total of 400 billion parameters. Quite a large model, but again, at any given moment only 17 billion of those 400 billion parameters are actually active. This model also apparently unlike llama4scout, is natively multimodal with a 1 million token context length. So already these models kind of aren't adding up here. These two here have the same exact active parameter count, but this one is way larger with way more experts, but yet has a worse context length. Whereas that's not the case with llama4scout. Already I'm starting to have some questions. Alright, then we move on to llama for behemoth. 288 billion active parameters across only 16 experts, but 2 trillion total parameters. So they call this the most intelligent teacher model for distillation. And we're assuming that both of these models are distilled from Llama 4 Behemoth. But again, the community is going to have something to say about that later. And this one is not released yet. This is just a preview. So what a massive model though, 2 trillion parameters is huge. I mean, I don't think there's any doubt in our minds that Behemoth must be pretty good diving a little bit deeper. Llama 4 Scout here they're claiming fits on a single Nvidia H100 GPU. And the way they phrase that sounds good. And then you look up how much vram an Nvidia H100 GPU has. It has 80 gigabytes. That's right. Llama 4's smallest model llama 4 Scout does not fit on consumer GPUs. Even the RTX 5050 90. So already a huge difference compared to not only the llama 3 series but also a bunch of competitors. I can run DeepSeek R1 on my RTX 5090, but not the smallest Llama 4 Scout model. They also claim again that industry leading context window of 10 million tokens which is huge like entire code base length. And with this context window they're also claiming that Llama 4 Scout delivers better performance than Gemma 3. Gemini 2.0 flashlight and and Mistral 3.1. Here are the benchmarks that Meta provides on their website for Llama Floor Maverick, the midsize model. And if we look at this benchmark, it's pretty clear that they're trying to position this as a wipe across the board, beating GPT4 Omni in pretty much every single benchmark. Same thing for Gemini 2.0 flash as well as Deep Seek v3.1. In this benchmark we're not seeing a single thinking model tested against it. We're also not seeing any of the top top models like 4.5 or 2.5 Pro Google. So if these benchmarks are to be believed in the first place, yeah, they're better than some of the more mid range size models, but they also don't run on consumer hardware. It's open source, which is great, but you still need actual server clusters like GPU clusters to run this thing for yourself. Now we move into needle and haystack. This is for long context. How good can it have successful retrieval over that context length? They are claiming a perfect score for a Llama 4 Scout here. 10 million tokens with 100% retrieval. Really hard to believe, especially considering I haven't seen any other AI company do something like this. But again, as I keep mentioning, the community is going to have something to say about this. Interestingly enough, Llama4Maverick, the larger model here, seems to do worse in their own benchmark. They claim it to be pretty great, but there's a couple of spots right here where it seems to screw up. Here's where things get really weird. At least on their own official website. We're looking at Llama 4 Scout, the smaller model, they're positioning this as a sweep across the board yet again, except for the fact that llama 3.70b actually scores a little bit better in the live code. Bench at 33.3 versus 32.8. Very close, but still seems to be losing a little bit. Finally, if we scroll down long enough we can see some benchmarks, some early benchmarks from Llama4 behemoth that is supposedly still true training. And finally we're seeing some comparisons to industry leading models, although notably Gemini 2.5 Pro is left out here and is kind of considered to be the best trending model. I'm wondering if they left 2.5 Pro out of the question here because it's already available and it probably beats Llama 4 Behemoth. Llama 4 Behemoth seems to stack up pretty evenly, maybe a little bit better than 3.7 sonnet, the older 2.0 Pro and GPT 4.5. All right, so now that I've got everyone caught up to speed on Meta's claims here regarding their brand new model release, I do want to just kind of solidify the VRAM requirements needed to actually run these models. Again, this is not even consumer grade hardware. The smallest quant here of llama4scout still needs 52 gigabytes of vram to run, so that'll fit on three 24 gigabyte GPUs at lower accuracy. Larger quants need at least 70 gigabytes. That is 1h100 as claimed directly in the blog post. If you want to run Maverick at all, looks like you're going to need at least 254 gigabytes of RAM, multi GPU, 832 gigabyte GPUs. I mean that's crazy. The target here for these models, if the VRAM requirements are anything to go by, are totally businesses and developers. No one has a machine, or almost no one has a machine at home that can run these models locally. And that is a huge step away from what we're traditionally used to from the Llama series of models. But it gets even crazier. Initial community reactions were strong. I mean we saw that 10 million token context window. Oh, this thing doesn't even run on consumer hardware. I don't care because it has insanely high context. This could make it a game changer for all kinds of different use cases that need tons of context. You know, upload your whole code base into the thing, upload books into the thing. Right? Sounds really great. As Always plenty of the liberators out here jailbreaking this thing immediately. So you already know that any safeguards Meta put in place were easily defeated by playing who uploads all of his jailbreaks publicly to GitHub. Then the vibe tests started to roll in. Jimmy Apples Great account to follow if you want to keep up to date with the AI world. And as far as the AI community goes, he's a high taste tester. His initial vibes on llama4 were meh. This was pretty surprising to hear. I didn't think that Meta wouldn't go down without a fight. Llama 4 should be a pretty good model, and it benchmarked, at least on Meta's own website, pretty well. And it wasn't just Jimmy Apples. There were a lot of people saying that something didn't feel right about the Llama release. So I decided to get a vibe check of my own. I ran this poll on X. Most people said that it had meh vibes versus good vibes. Still some decent good vibes though, in total. But I also ran this same exact poll on my YouTube community with 800 total votes and the trend actually gets worse with more votes. More people say it has meh vibes, less people say it has good vibes. And obviously, you know, these aren't scientific, but I just wanted a little temperature reading from the community. Now since the vibes were so off from the community, I wanted to see what their tests were entailing and what the community has found actually using these Llama 4 models for themselves. First up, we've got a test from Flavio here, a phenomenal account by the way, in the AI space to follow this is a coding demo between a couple of different models. Obviously we want these balls to be bouncing through this little physics arena physics maze in a realistic way. You can see Gemini 2.5 Pro at the top nails it. But that's to be expected because it's kind of the industry leading model right now. People consider it to be the best, especially with coding. So no shock there that Gemini 2.5 Pro nails this current GPT4O the new variant. In quotes. This model also does a pretty phenomenal job. The bounces are pretty darn good, but the balls seem to maybe disappear a little bit over time or they get mushed around. Sometimes they fall through. So it's not Perfect. Here is llama4 and I know it doesn't say the specific model version, but this is Maverick double check for you guys. You can see it's kind of all over the the map, the actual maze lines are contradicting and touching each other. It's not nearly as cohesive as 2.5 Pro. The balls are sort of glitching around, they're not bouncing realistically at all, and they eventually kind of fall out of the whole piece. GPT4 omni old though, does even worse than llama4maverick, suggesting that this is maybe the original benchmark that Meta was trying to beat with Llama 4. But the AI space moves really quick and OpenAI already has a new GPT4O variant out that does better in this test. As Flavio points out, this thing is free open source and Pretty close to GPT4O pre update if you think about it, it is open source, but it doesn't run on consumer grade hardware. It is free, but. But I can also access GPT4OMNI for free through ChatGPT. I get it. It's supposed to be targeting developers, it's supposed to be targeting businesses. But Llama 4 isn't the only open source model on the market. We've got deepseek R1 now we're OpenAI is even planning to release their own open source, fully open weight model pretty soon and we probably can assume that it's going to be at least as good as GPT4.0 new, right? So I don't know. As we learn more, the positioning for llama 4 becomes increasingly more tricky. But it doesn't just stop there. All right, let's take a look at this benchmark from the community. This judges long form creative writing, long form meaning long context. Keep that in mind. And of course, creative writing, not the easiest task for AI models. All right, at the top of this benchmark we've got Gemini 2.5 Pro. Obviously, like I said, this thing's kind of the king right now. It's got kind of a high slop rating at 40 and if we actually click on the information we can see the top bigrams repetitive words and such that it's using as it's being benchmarked. The repetition though is really low at only 7.3 and the degradation over here is only 0.15. So you can see this line is pretty flat. As the context length increases, it doesn't degrade too too much. This gives it an overall score on this bench of 80%. And again, this is top performing. Deepseek v3 is right behind it at 78.1. Little bit more repetition, little bit more slop, but the derogation is pretty great For Deep Seek v3 3.7 Sonnet's also on here. The latest GPT4O is also on here from 327. Pretty low repetition at 9.4. Low degradation even lower than Gemini 2.5 Pro 76.8 total score. Move down to Gemma 3. You can see the dedication is all the way up at 0.9 for this model. So you can see the long context. It's really suffering over time with more slop and more repetition. But let's jump all the way down to llama 4 Maverick. This brand new model slop 74.6. That's really high up. You can see all of the top repetitive words glinting twinkled top bigrams and trigrams. Repetition is all the way up to 40. Keep in mind current GBP GPT4O is only at 9.4. Don't forget Deepseek R1 is also only at 8.5 and Deepseek V3 is at 11. So we're seeing worse repetition and the degradation looks pretty terrible actually. 0.627. It's really a downward curve here. Other models like GPT4O Claude 3.7 Sonnet Deepseek V3 these models are holding their own in degradation over time. Long context is not working out for long form creative writing here for the Llama 4 series of models whatsoever. It's even worse for Llama 4 Scout as well. Slop rating of 80 which is really high. Terrible degradation over time and tons of repetition. These scores are noticeably lower. This separate community benchmark is really showing that Llama 4 might be designed or trained to do well on certain benchmarks. But benchmarks it's not trained on like this one. It shows its true colors. And it's not just this one as well. Move on to creative writing again. We have Deepseek R1 at the top. GPT4O latest also up there. Gemini 2.5 Pro. As we'd expect, scroll down to find Llama 4 Maverick not doing nearly as well. Overall score of 800 versus Deep Seek V3's 1373 and Llama 4 Scout does even worse at 710. It's not looking great. We're barely doing two times better than GPT 3.5 Turbo. Colomze here ran some quick tests asking for a specific quote. Claude 3.5 haiku gets this correct. Llama 4 Maverick gets this wrong. And it's initial tests like these combined with these community benchmarks that have led people to kind of declare Llama 4 as a lost cause. It's over. It doesn't even run on consumer hardware and it's not really benching in reality. It's not really tasting like Meta told us it was going to taste. But it gets even crazier, guys. Now this I take with a grain of salt. All right, it is an anonymous post from the local Llama subreddit. Serious issues in llama4training and this particular person claims to have submitted their resignation. Listen to this. Despite repeated training efforts, the internal model's performance still falls short of open source state of the art benchmarks lagging significantly behind this right here. This statement lines very much with the EQ benchmarks we were looking at and not the official benchmarks from the Meta website. Furthermore, it goes on to say that company leadership suggested blending test sets from various benchmarks during the post training process, aiming to meet the targets across various metrics and produce a presentable result. This is what they would call cooking or nerfing the post training process to lead to better benchmarks. Is this what actually happened? If this claim is true, if they suggested this and actually did blend the test sets from various benchmarks into the training data, well then these benchmarks are negatable. They are gamed. You're supposed to train these things blind and see if they can complete the benchmarks, not train them on the benchmarks specifically. Now again, these claims, they are not from Meta themselves. They are not very verifiable. It's an anonymous post on Reddit. But if this is true, then that's not a good sign. If we read on failure to achieve this goal of benchmarks by end of April deadline would lead to dire consequences. Following Yesterday's release of Llama 4, many users on X and Reddit have already reported extremely poor real world test results. So this person finds this approach unacceptable. They have submitted their resignation and explicitly requested that their name be excluded from the technical report of llama 4. The VP of AI at Meta also resigned for similar reasons. And of course once this post started, circulating around the community is like, alright, if Meta actually did this for llama4 training to maximize benchmark scores, it's totally screwed. The model is worthless. I don't think anyone wants to use a model where we don't even really understand how it benches and the benchmarks that the company is showing us are gamed or lied. Chubby here, A prominent community member who was previously pretty excited about the release of LLAMA 4 notes that if this letter is true, if this message is true that Meta cheated on benchmarks, it would be an unprecedented image loss for Meta. And I totally agree, Meta already doesn't have a great image in general I think, and they were significantly improving it with the Llama series of models. Llama 1, 2 and 3 were all bangers, but Llama 4 here, I mean if you guys are cheating on benchmarks, how are we supposed to trust you? And the evidence really seems to be pouring in because we have more benchmarks from the community that are super suspect and really confusing. We've got a Misguided attention eval. Llama4Scout performing at 35.9%, significantly beating out Llama4Maverick at only 29.5%. Llama4Maverick is supposed to be better and still they're losing quite heavily to Deep Seek R1 and Deepseek V3. And I thought that this community post had some really interesting insights as well. Chase Brower here notes that Maverick and Scout seem to be completely different lines. Scout is its own thing, pre trained by itself, while Maverick is co distilled from behemoth. Chase here guesses that Maverick is pure panic model trained in the last couple of months. This might be the model that's actually post trained on the benchmarks, meaning that Scout might straight up be a better model if it was truly pre trained by itself. Chubby posts Another benchmark, this time Fiction Live Bench saying that the original claimed 10 million context window does not look so good. In reality, with Llama 4 Maverick and Scout both faltering at even low Context windows of 400, 1000, 2000, 400010 million context window doesn't really seem to be real when these benchmarks that we're running have these two models both falling flat on their face and performing worse than all of the other competitors they claim to be better than. Here is EITR Polygot Benchmark Llama4 Maverick Doing pretty poorly here. Sure the cost is very low, but Gemini 2.5 Pro and Claude 3.7 are stomping all over it. Deepseek V3 is clearly ahead by orders of magnitude. Techacon Cheer says that it's no wonder Yann LeCun says AGI won't arrive this decade when metas are not really cooking too well in the background here, and in some cases seem to be cooking their benchmarks to look better than competitors like Deep Seek v3, when in reality they actually aren't. Finally we come down to this. This is a leak from two months ago that might actually have some truth now that llama 4 is out. All right, here's the original leak. It started with Deep Seek v3, which rendered the Llama 4, which I assume was training already behind in benchmarks. Engineers at Meta moved frantically to dissect Deepseek and copy anything and everything that they can from it. Not even exaggerating apparently from this leaker. We always have to take this with a grain of salt. We don't know if it's real, but this was two months ago. Management is worried about justifying the massive cost of Gen AI. How would they face the leadership when every single leader of Gen AI is making more than what it costs to train deep seek v3 entirely? They have dozens of such quote unquote leaders. Deep Seek R1 made things even scarier. Meta's generative AI should have been an engineering focused small organization, but since a bunch of people wanted to join the impact grab and artificially inflate hiring in the organization, everyone loses. And that's sort of how the downfall of Llama 4 went on, so to speak. This might very well be true because llama 4 is disappointing and I'm I'm really sad with the results the community has found so far. I had high hopes for llama4. I thought meta had it on lock. So yeah guys, that was the Llama 4 launch and the slow descent into what appears to be really tragic launch for Meta. I had high hopes for this model, I really did. The last three were were totally awesome and competitive. I mean we have no reasoning models from Meta at all here. The new Llama 4 doesn't even run on consumer grade hardware and it seems like they gamed their own benchmarks potentially with real world tests falling extremely flat. The light testing that I did do with the llama 4 wasn't really impressive. I prefer GPT4 omni, the free model and chat GPT. I don't know what to say to Meta other than to get it in shape. I mean if you guys really did cheat on the benchmarks, come clean about it, fire everyone who was involved and start from scratch there. Hopefully more about this kind of comes out. As the week proceeds we'll see some more test results. For now though, Llama 4 is kind of sitting on the back burner in my mind. I'm more focused on the new open source releases that are supposed to come from OpenAI this year.04 mini and full O3. Not to mention Google's got Gemini 2.5 Pro which really seems to be a strong model. But that's it from me for today. Enjoy your week folks. And yeah, let me know what you think about all this. This is absolutely craz. See you in the next one. Goodbye.