New OPEN SOURCE AI Just STUNNED The Entire Industry (Beats Everything!)

So in this video, this is probably going to be the largest announcement in terms of open source AI. And I think the implications for this model are truly staggering. This video is introducing the Reflection 70B model which is the world's top open source model with only 70 billion parameters. Now you might be thinking, okay, open source is great, that's fun and games, but, but state of the art closed Source models like GPT4O, Google Gemini and Claude 3.5 Sonnet are just far ahead of where open source models are. But that isn't true anymore. After Matt Schumer has produced this fine tuned version of Llama's 3.170 billion parameter model, we now have a state of the art system that many individuals can use for free. So this model is incredible. Let's take a look at the benchmarks to show you all why this announcement absolutely went viral. So we can see here that on the left hand side we see reflection 70B and we can see that it currently only loses in two benchmark areas, the Human evaluation and the gpqa. Now what's crazy about this is that it doesn't even lose by a long shot. It only loses by a few percentage pieces to Claude 3.5 Sonnet, which is arguably the strongest model on the planet all around. And if you've used Claude 3.5 Sonnet, you'll know exactly how capable that model truly is. Now what's incredible about all of this is that the MMLU, it surpasses the math benchmark, it surpasses the GSM8K, it surpasses the and the IF eval, it surpasses. Now the reason that this is so incredible is because in the fine print details, a lot of the times what people miss is they miss the ability to see how these models are actually responding to these prompts. So what we can see here is that anytime there are benchmarks for an AI system, what will happen is they will ask the model questions in certain ways. For example, for Claude 3.5 sonnet, when they ask them this question, this was a zero shot chain of thought, which essentially means that you ask the model a question and in doing so, you ask it to lay out its reasoning steps so that it can come to a better response. And oftentimes when we see certain benchmarks broken in brackets, you should be paying attention to whatever it says, because this is how they achieve that number. And many a time what we will have is we'll have new large language models or omni models or systems being able to hit the number one level in terms of the benchmarks. But what you won't notice is you won't notice the benchmarks being changed due to the text in brackets saying a specific way of responding. For example, right here you can see that Gemini's MMLU score is rather good, but it is based on a five shot response. It's meaning that five examples were submitted first to the model and then it was given the actual question in order to respond. We can see that the same thing is there for Claude 3 opus 5 shot brings it up to 86%, whereas 0 shot brings it down to 85%. Now, what's crazy about all of this is that reflection 70B we can see that the 0 shot, which is where you just simply ask the model the question, it manages to surpass all of these other models. So we can see zero shot reflection manages to do exceedingly well under a wide range of scenarios, leading us to believe that this model is currently state of the art in terms of not only open source, but potentially nearly closed source. Now, later on we're going to do some individual tests, but I think this is one thing that is truly fascinating. I mean, for example, on the GSM 8K, we can see that this is 11 shot and, and this one is 8 shot. So truly, truly remarkable stuff. Now, if you're wondering how these models actually work in terms of how they manage to get their responses down, we can take this example here. So we've got the question which number is large, 9.11 or 9.9? And then the AI has its thinking. Now, essentially this is where the AI thinks about the problem. So it says this is a relatively simple problem that involves comparing two decimal numbers. I can solve this using a straightforward approach with chain of thought reasoning. So reflection 70B it uses chain of thought first to plan out its action. So first we can see the plan which is to firstly identify the numbers and compare 911 and 9.9. Compare the number whole parts, if whole number parts are equal, compare the decimal parts, then determine which number is larger based on the comparison. And, and then now you can see after the plan, it then executes this plan with the chain of thought. So after the plan we have the chain of thought right here. And then it goes through step by step and then it comes to its solution. So this is part one, I'm just going to do this part one here where we've got this first bit. Then you can see right here it goes down to the actual chain of thought, which is the part two where it reasons through the actual steps above. And then this is where the genius of reflection comes in. This is where we have the actual reflection. And this is where the model checks and reflects back on its responses. So this is stage three in terms of the model responses. And this is where it says checking my reasoning. The approach is correct for comparing decimal numbers. The comparison of whole number and decimal parts is accurate. The conclusion is logical. Based on the comparison, the the reasoning appears to be sound. No corrections are needed. And then after these steps, step 1, step 2 and then step 3, we get the final response of the output. 9.9 is a larger number than 9.11. Now the reason that this is rather important is because many a time some of the models have struggled with this simple reasoning problem. And this problem question would was used to highlight the fact that these models may struggle with certain reasoning capability. Now there was a paper released in 2023 that did talk about reflection language agents with verbal reinforcement learning. And this was something that was quite similar to verbally reflect on task feedback signals and then maintain their own reflective text and an episodic memory buffer to induce better decision making in subsequent trials. So this is something that showed remarkable improvement when they got these models to reflect back on their tasks. Now you might be thinking, okay, this looks amazing, but how does this work in actual real world scenarios? So I'm going to show you guys a real chat that I just had based on a different benchmark that many people don't use. So this is the simple benchmark, which is simple and basic reasoning that tests reasoning in a variety of different areas. Now if you click the try it yourself, you can see that these are the kinds of questions that we do get. These questions aren't difficult for humans. If you actually read them and do them yourself, they're not that hard for humans. If we go to the leaderboard, you can see that the human average is around 92% where other models actually struggle at around 27% and 5% for GPT4O mini. So there is a huge gap between the human best and and the current best state of the art system. Now this doesn't mean these models are dumb, it just means that these models do struggle with basic reasoning. Now what I've done here is I've input some of these questions into the Reflection 70B model because I wanted to see how its reasoning steps managed to transfer over to its new thought process, chain of thought and of course entire reasoning process. So basically there's this question here about cookies and the answer is C. So it talks about different cookies. It says on a table there is a blue cookie, a yellow cookie and an orange cookie. And I've drawn three cookies here for visual demonstration. It then says, a purple cookie is placed to the left of the orange cookie. So we can place that purple cookie right here. I don't know if you guys can see that. Then it says, while a white cookie is placed to the right of the blue cookie. And then I can place a white cookie right here, which isn't too accurate, but it doesn't really matter. Then it says the blue hatted girl eats the blue cookie, so we can cross off that cookie. Then the yellow hatted girl eats the yellow cookie and three other cookies. Okay, that's what it says. And then 1, 2, 3. This now means that all cookies have been eaten. And then it says, so the last orange hatted girl will be unable to eat a cookie because there's none left. Now, interestingly enough, when I submitted this question, unfortunately the model actually does get this wrong. It says the most likely scenario is D, the girl eats just one or two cookies. This is because option B is impossible. Option C is unlikely, given that the girls are described as bored and cookies are available. But the point is, I'm not going to go through all of this. But what we do have here is the situation that it didn't get this one right. Now, on the other question, the question about ice cubes being placed into a frying pan at the start of the first minute. And it then has some information that is meant to distract the model. And then it basically asks how many whole ice cubes can be found in the pan at the end of the third minute. You can see that the easy answer is zero because they are ice cubes and ice melt. It will melt if it's placed inside of a frying pan. And it says, please give the most realistic option. We can see that it's reasoning steps. It says, let's approach this problem step by step, talks about how the ice is placed. Then of course it does the reflection. And interestingly enough, you can see here that it actually says, wait, I made a mistake in my reasoning. The question asks for the number of whole ice cubes that can be found at the pan at the end of the third minute. I forgot to consider that ice cubes melt quickly in a hot frying pan, especially if it's frying an egg. It's unlikely that all ice cubes would remain whole and unmelted after being in the hot pan for several minutes, given that the pan is frying a crispy egg, it's at least likely that some of the ice cubes will have melted partially by the end of the third minute. The most realistic answer is a number that would be less than 20. So you can see its answer here was 20 ice cubes in the pan, which is of course wrong. And then you can see it then says okay, I'm ending my thinking right there. That means that the most realistic answer is option C. It says given that the pan is frying a crispy egg, which indicates a relatively high temperature, the most realistic answer option is C0. This doesn't mean they've all necessarily melted, but rather than none of them would remain whole after being in a hot pan after three minutes while an egg is being fried. And the other options assume that the ice cube would remain whole, which is unlikely in this context. And I think this shows us why this is so powerful, because the model would get this wrong initially if we just asked the model zero shot with the base model. But since this model was fine tuned to have this step by step thought process, we can see that this model can clearly and easily identify its mistakes. Now, funnily enough, I did try this model out and it did perform really well it in many different reasoning areas. But I wanted to see how other models do perform on these same questions. Interestingly enough, Claude 3 Opus does actually get it right. It says the correct answer is C be unable to eat a cookie because all of the cookies have been eaten by the other two girls. And for the ice one, you can see that it also gets it right, which is C. You can see that it says even though 20 ice cubes were added in total, they would have all melted in the hot frying pan by the end of the third minute. Now, I asked the new version of Gemini and surprisingly it also got this right. You can see that it says here that the most realistic answer since the ice cubes are in a frying pan, the most realistic answer is C0. And then it says here for the other question that the most logical answer is C be unable to eat a cookie. So it shows us that these models are getting increasingly smarter. Now, now, I do want to note that this is Gemini's experimental model, which does mean that this is likely Google's most advanced reasoning model in terms of its capabilities. Because I know Google have been desperately trying to improve the capabilities of their model. And with Claude 3 Opus, this model is just remarkably good at reasoning. Now, it's only two questions, so it is not a huge sample size. What I would have to do is somehow access a lot more of the questions and to see just how good reflection 70B is. Or what we would have to do is actually take a look at the SEAL leaderboards. These are expert driven private evaluation. They're quite different to the traditional evaluations because they use private data sets. So these can't be gamed like you can't train on them. And these ensure unbiased and uncontaminated results. There's of course expert evaluations using domain specific methodologies ensuring the highest quality and reliability. Now you can see here on this we have different areas such as coding, adversarial robustness, instruction following and math. So I would like to see where this 70B model does perform. But I think one of the main things here is that even if this model is is second or third, I think a 70 billion parameter model forming at state of the art and giving them a run for their money is something that I genuinely didn't think would happen. I thought that the open source to closed source gap would be a lot further ahead. But I guess that since these open source models don't have to do things like six months of safety testing means that they can quickly output different iterations of the models so we can see exactly how things are tested. So with that being said, let me know what you think about this model. Is this going to be something that you saw coming? Are you surprised by this at all? And what are your thoughts on the state of open source? I think this is a remarkable jump and I can't wait to see the 405 billion parameter model to see how it performs against closed source.