Open AI's New Model Is Finally Here.... (Strawberry /Star)

So there is a lot of speculation going on around OpenAI's strawberry model. If you aren't familiar with the Strawberry model, it's a model that is supposed to be the next stage in terms of AI evolution. And what I mean by that is that these models are able to reason in ways that are at human level, which gives them the ability to output responses that are what humans would do. And this just basically means that these models are going to be a complete level above the types of reasoning that we are used to. So I'm going to dive into some of the information, and then I'm going to dive into testing of OpenAI's Strawberry model to show you some of the interesting things that I've managed to find. So one of the things that we do know here is that there has been a lot of hype around this model. There's been a lot of, you know, questions and things and tweets. And one of the things that has spurred a lot of this recently was Sam Altman himself. So if you've been on Twitter and if you aren't on Twitter, I would say it's definitely the space to be if you're trying to actually keep up with some of the AI updates, because sometimes you do miss some things, sometimes I don't cover them in videos. But if you want to know about all of these smaller updates, it's definitely a good place to be. Now, what Sam Altman tweeted here was pretty funny because he tweeted, I love summer in the garden. Of course, you can see in this image, this is a image of his garden, which has plants that are growing strawberries. Now, from this image, it's timed pretty perfectly because around the same time, there were other accounts talking about strawberries. Now, this post did get a lot of attention. It got like 1.5 million views, 10,000 likes, 1,000 retweets. And this, of course, is a clear reference to OpenAI's Strawberry model. It was Reuters that previously reported on the Strawberry model, which was the new name for the previous model, the Q1. So when Sam Altman actually tweeted this image, a lot of people were speculating, and they're still speculating that we might get a new model release. Now, what was interesting is that at the same time that Sam Altman was tweeting this, there was also this account that was posting on Twitter quite a lot of tweets. In fact, so many tweets per minute that it seemed as if it was some kind of advanced AI agent that was posting It. And although there were a million different posts by this account, I do think it was a person. Some people do think that this Strawberry themed account was just a random person. But over time it became clearer and clearer and clearer that this account did have some OpenAI affiliation. So what this account was doing is that it was posting many, many different tweets. But one of the most interesting tweets that it recently posted kind of gave me the inclination that this was a little bit bigger than I may have initially thought. And that tweet was this right here. So this tweet says, welcome to level two. How do you feel? Did I make you feel? And then the craziest thing about this was that Sam Altman actually responded to this tweet and he says, amazing. Tbh. So the reason that this was pretty crazy was because, of course, first of all, Sam Altman responded to this tweet and there were many people that were essentially dismissing the claims about this account and basically just stating that, yo, look, this is just a random post and they're, you know, just Shizzo posting or whatever on Twitter. It's not anything to do with actual Sam Altman or the company, but clearly Sam Altman responding to such a small account that only recently started posting definitely makes this situation a lot more interesting. Now, what's far more interesting than that was this tweet. So many people did actually miss this tweet right here. But the welcome to Level two thing is actually a big thing because recently, like I said, there was all of this information regarding how OpenAI is tracking their progress towards AGI. One of the things they've been doing is, is they've been setting out these five levels. Levels range from chatbots all the way up to organizations that can do the work of an organization, essentially an autonomous AI agent company. Now, essentially, the welcome to Level two is referring to this right here. You can see that it says level two reasoners that have human level problem solving. And if they're stating that, welcome to level two, which is what this account said, and then when we look back at this, you can see that Sam Altman responded saying, saying, did I make you feel? How do you feel? This feels amazing. It's clear that basically the message that we're getting is that because previously OpenAI said that they were nearing level 2 reasoners, which is human level problem solving, that maybe with Project Strawberry, now they have cracked the reasoning thing and now that they are at human level problem solving. So that was the kind of information that we got there because it's like, whoa, okay, if we now have human level problem solving, then this is a really big deal. And this is quite incredible for us. So that was why this tweet was the one that I screenshotted and I put here, because if we are at level two, which means, you know, right now we have human level problem solving, this is a big deal. The reason this is a big deal is because if you look at what Reuters said, Reuters said that OpenAI executives told employees that the company believes it is currently on the first level, but is on the cusp of reaching the second, which it calls reason. And this refers to systems that can do basic problem solving tasks, as well as a human with a doctorate level education who doesn't have access to any tools. So this is pretty incredible because not only is it human problem solving, but it also is that without any tools. So this would mark a really, really incredible milestone in terms of the AGI levels, because this would mean that we're at level two and that all that would need to happen now is that level three, which would be next year. And we already know that in another video, I'm going to talk about this, but there's a lot of speculation regarding the complexity of these models and the types of things that they're going to be able to do next year. Like, everyone's got their eyes on 2025, so I think by far that's probably going to be one of the most impactful years of AI development. Now there's some more information, because if we actually take a look at some of the original articles for Strawberry, just to jog your memory, there's some pretty crazy stuff here, because it says that one of the main things about Strawberry, it says, it describes a project that uses Strawberry models with the aim of enabling the company's AI to not just generate answers to queries, but to plan ahead enough to navigate the Internet autonomously and to perform more OpenAI terms of deep research, according to the source. And this is something that has eluded AI models to date, because, of course, as you know, getting AIs to perform an extended chain of actions over time is quite difficult because what you need to have is you need to have a higher rate of reliability because. Because if you make an error in the first stage, the chances of you making an error in the second stage, the error rate just continually just goes up if the error rate is not sufficiently small enough in the beginning. So high reliability and long chaining actions is something that has long eluded current models. So if Strawberry is As good as they say it is, then this would be a complete breakthrough. Now, of course, you can see right here that it talks about how last year we were basically talking about how this progress in terms of AI. One of the biggest things that we wanted was for AI to be able to increase its reasoning ability. And this was something that was spoken about in a lot of detail. Now, all of these things are very important and all of them are very fascinating in terms of what we're looking at in terms of reasoning ability, being able to plan ahead, reflect how the physical world functions and work and stuff like that. And of course, them Sam Altman tweeting this Strawberry thing, then of course, this account, this random account popping up and saying, welcome to level two, we can kind of guess that, okay, we might be on reasoners, which are human level, problem solved. Now I'm thinking, okay, how do we even know this is the truth? Because now this account earlier, just two hours ago, tweeted, this is available for direct chat. Enjoy. So this chatbot right here that you can see is SUS column R. And SUS column R is basically, I'm guessing that this is the model, which is basically the OpenAI model. So this is basically the strawberry model. So sus column R, SAS column relating to the fact that Strawberry was a problem where AI models couldn't count the number of letters in the word Strawberry. And I did some recent testing on some questions that are common sense reasoning problems. And when I was doing this, this video took a lot longer to make than I initially thought because I was testing across a variety of different models. Gemini 1.5 Pro experimental, the Strawberry model, and initially Claude 3.5. And I found some rather strange things that I do want to share with you all, but I do think that this is something that needs further investigation. Now, I am currently rate limited, which essentially just means that I've used the model too much. So I'm going to have to wait for a few more hours and then I'm going to get back on it. But one of the first questions that I wanted to test, and this was a question from AI Explains Bench, was essentially this question right here. So let me actually find the question, because now I'm just looking at the answers. But the question was, Beth places four whole ice cubes in a fire at the start of the first minute, then five at the start of the second minute, and then some more at the start of the third minute, but none in the fourth minute. If the average number of ice cubes per minute placed in the Fire was five. How many whole ice cubes can be found in the fire at the end of the minute? Basically, the answer to this question is that there's going to be no ice cubes because you placed ice cubes in four fire. Obviously they're going to melt. And here's the thing, okay? Gemini 1.5 Pro got this wrong. It said 5 when looking at GPT4O. It got this wrong. It said 11. You can see right there. I've highlighted it in 11. It's decided to ramble on and just completely ignore this stuff. And interestingly enough, when I looked at Strawberry, it did manage to get this question right. Okay? It actually managed to get this answer right and quickly. For those of you who want your beloved Claude 3.5 sonnet, I really do like it, but it got it wrong. It said the correct answer is D, which is 20. It just didn't understand this question, I guess. But one of the things I actually really did like about the way that Strawberry decided to answer this question, it said that. It said, thus, the most realistic answer of whole ice cubes still in the fire at the end of the third minute would be zero. And this assumes that the whole ice cubes imply those that have not melted us whole, which in a fire would be none by the end of three minutes. However, if we interpret whole more leniently to include those that are not completely melted, then 11 could also be considered, since those placed in the third minute would have not had time to melt significantly, given the options in the realistic scenario in the fire, B might be a better choice if we're counting cubes that are still largely intact, which is out of all the models, it's by far the most detailed answer and of course the most correct. But what was weird about this entire thing is that Gemini 1.5 Pro just managed to, you know, just like that, it managed to get this answer correctly. It says the most realistic answer is zero. Ice melts quickly in fire. And the question asks about the end of the third minute, and by this point, ice would have melted completely. So, I mean, Gemini 1.5 Pro just completely aced this. But after realizing that there were inconsistencies with the benchmark, and I was like, okay, Strawberry might be a bit better in terms of the reasoning ability. What I wanted to do was I wanted to test this. So I looked at this paper called easy problems that LLMs get wrong. Okay? And this says we introduce a comprehensive linguistic benchmark designed to evaluate the limitations of large language models in domains such as logical Research reasoning, spatial intelligence and linguistic understanding, among others. And through a series of straightforward questions, it uncovers the significant limitations of well regarded models to perform tasks that human manage with ease. The point here is the reason that I've chosen this paper is because humans can reason about these questions with relative ease. Okay. And once you understand that these are questions that humans can do with ease, you're going to see that how LLMs are reasoning about these problems is a bit different. And I think that the way that these, either the system prompt or the method that they're trained kind of can change different things. Okay. Okay, so let me show you some of the questions that were from this. Okay. And then I then tested them on Strawberry. So let's go through these right now. So you can see right here that I said you're in a room with two doors that lead out. And these are the questions from the paper. But anyways, let me just read it. So you're in a room with two doors that lead out. One door leads to certain death and the other door leads to freedom. And now there are two guardians, one by each door. One taller guardian always tells the truth and guards the death door. Okay. And the other one always lies. What is the minimum number of questions needed to ask the guards to get out to safety? The answer to this question, it's fairly straightforward, because if there is one taller guardian that always tells the truth, and there is, you know, another door that always lies, and you know it's the death door, you can just simply pick the shorter door and you'll know that that is the door to freedom because the taller one is guarding the death door. So you, you don't really need to ask any questions. But the problem is that Strawberry just rambles on about many different things and it doesn't get this answer right. Now, crazy thing about this is that none of the other models do get this question right. So when I was trying to test this out against all the other models, none of them managed to get this question right, which was rather fascinating because humans on this question, they actually managed to get this question right quite a lot, but there's still a little bit more. So I then asked it another question here, and this was another simple one. I said, how many pairs of twins do you need in a room for there to be at least a 50% chance that two people had the same birthday? This question is really straightforward. You only need one pair of twins because twins are born on the same day, unless of course, you know, they are born like minutes between each Other, but the problem is, is that, like, you know, of course twins can be born. Like, you know, if someone's born on like, January 31st at like 11pm and then one's born at like, January 1st of the next year at like, you know, 1am or whatever, they're going to have different birthdays. But the problem is that the question didn't consider this and it immediately went to the birthday problem, the birthday paradox. And other models did the same as well. Okay. And I started to notice a trend when I was looking at how Strawberry was reasoning around these problems. We got another one here as well. And this one really surprised me because I was like, okay, what is going on here? This doesn't even make sense. Okay. Like, for example, I said, count the number of occurrences that you know, of the letter L in the word lollapalooza, and it said five. And I asked this question to the model twice, okay? And it said the word L appears in the word Lollapalooza five times. And that's completely wrong. We've got 1, 2, 3 and 4. So it's just 4. And Claude managed to get this right, and Gemini 1.5, the standard one, managed to get this right as well. So, which is remarkably weird. So I started to think, and I know you guys might be thinking, okay, this is supposed to be OpenAI's flagship reasoning model, which is Strawberry, and it's supposed to have human level reasoning, but a human would be able to get this really easily. Like, if these other models can get this really quickly, why can this model struggle to get this basic question? So simply, number one is probably because of how it's trained, and number two is because I think that this Strawberry model, I think that the reasoning ability might be good, but I think what I've done here is that I might be testing it on the run benchmark. And I'm going to elaborate on that later on in the video. Depending on this isn't just a cop out for OpenAI, it should be able to get this question right. But I think when we look back at what OpenAI have tried to achieve with this model, I think we do need to have benchmarks that accurately reflect what that model is trying to achieve. Now, there was one more question I wanted to ask and I asked this question because Once again, Gemini 1.5 Pro got this question right. And it was weird because no other model managed to get this right other than Gemini 1.5 Pro. And that's their basic model that, strangely enough, struggles with any other tasks. And I'm going to come to a conclusion in just a minute. But I said a Runway trolley is heading down the tracks away from five people upwards of the track. So a Runway trolley is heading down the tracks away from five people upwards of the track. You are near a lever that can switch the trolley to another track. Does it impact people's lives if you pull the lever? Obviously it doesn't because it's going, you know, upwards and away from the track, which is clear here. And you can see that Gemini 1.5 Pro says it's already moving away from the people. So it's not going to change that. But Strawberry, once again, it manages to say yada, yada, yada, it manages to just ramble, ramble on. And I think this is indicative of a wider conclusion. Okay, this is the conclusion that I came to is that Strawberry might be a good reasoning model, but I think it's trained to reason in multiple steps about every single problems and then of course, thus overcomplicating things. And I think that is going to be an issue here because other questions that I've asked to other models, they don't get confused like this. So this is one of the questions that I've said here that how do you measure exactly 4 gallons of water with only a 3 gallon, 5 gallon and 4 gallon jug? And then you can see here that it goes on this complete ramble about, fill the five gallon jug, then do this at three gallon one, then basically you've already got the four gallon jug, just put the water in it and measure it. But interestingly enough is that at the end it says you could start with just filling the four gallon jug to get gallons directly. If you interpret the question as having a four gallon jug for measurement rather than just for holding water, however, this approach skips the puzzle's intended solution method. So right there you can see that Strawberry thinks that the answer is too easy. So it thinks that, hey, look, the answer is too easy. The puzzle's intended solution method is to of course reason throughout the problem and then of course do that. Which doesn't mean that the muzzle is stupid and it just means that what it's chosen to do isn't correct. And I don't think that that means that the model has limited intelligence. I just think it's a bit of a little bit confusion about what can go on here. Because I think you would rather a model be able to completely understand how to reason through all the steps with two different jugs, rather than not realize that this is a Trick question. So I think when we look at models like Gemini, and the conclusion that I have with Gemini is that when I put it through all of these questions, the ones that Strawberry, a lot of them got wrong. What I found interestingly was that Gemini always said that this is a trick question. Okay? Gemini was like, hey, this is a trick question. It's designed to make you think about the complex setup while the answer is straightforward. So I think the interesting thing here is that Gemini, it didn't reason much. It was just like, look, this is the answer. But with Strawberry, it managed to reason a lot. Now, what this makes me think is two additional things. Number one, this model is probably trained to reason when system prompt is like to reason step by step through a million different things. And because of that, I think that this model might not be great for testing on certain problems like this, because these questions don't really test your reasoning ability. They test your ability to identify trick questions. And I think that's not the best benchmark. So what I'm going to do over the next couple of days is I'm going to test these on a completely different benchmark that actually where the actual question requires long level reasoning. And I think that would be more important because these trick questions where LLMs don't. Don't identify them, yes, you could say it's supposed to, but it doesn't really test how well this model would perform in cases where long level reasoning is required. And one of the things that OpenAI did say, which is why I brought this article back in, was that they actually did say that one of the things that they wanted to do, okay, and this is the wrong slide, this slide right here was that it wanted to do plan ahead enough to navigate the Internet autonomously. And I think that I don't know how you test that with just a simple chatbot. Maybe you could ask it questions on how it would do different things. Things which is, you know, rather fascinating. And I think that's rather important. Now, there was one thing that I did find rather impressive because this is the question the Strawberry got right that nobody else managed to get right. So there was a question that was, if I can find the question, I'm going to just quickly find it. But ask this question. If I walk to my friend's house averaging 3 miles per hour, how fast would I have to run back to double my average speed for the entire trip? So essentially, we got this question from Strawberry, and basically, if you just think about it like this, so if, imagine, you know, you're walking to your friend's house at 3 miles per hour, you know, if it takes you one hour to get there, you've traveled three miles in one hour. Now if you want to run back home so that your average speed for the entire trip is 6 miles per hour, for your average speed to be 6 miles per hour for the entire trip, you would have to cover the entire distance, which is six miles, which is, you know, three miles to your friend's house and three miles back in one hour. But you've already done three miles in one hour. So it's pretty much impossible. And it does get this right through its reasoning, you know, capabilities here. But overall I think what we have here is a situation that I think this Drawberry model is a little bit strange in the sense that, you know, it does get some of these questions wrong. You know, like shockingly it does get some of these questions wrong. But what I will say is that I'm not putting this model off just yet simply because I do think that there are new benchmarks needed to test this kind of model because the kind of, you know, calculations it does are a little bit different compared to these other models that didn't get the question right. So it definitely does get certain questions right that other models don't in one off scenarios. But there isn't enough evidence for me to say that this model is definitively better or definitively worse than frontier models. But I got to be honest guys. So yeah, it's definitely a kind of interesting thing here. But I do think that we do need a different benchmark because all of the tests that I've seen online of people asking different questions and stuff like that, I don't think they're that useful because these are not the kind of questions that you do get in, you know, day to day. And I don't think they're the kind of questions that would be useful for practical applications. Like for example, if we want a model that can go out and do deep research, of course you don't want it to get stumped by very simple things. But you aren't going to be testing these models on whether or not it can identify a trick question. You're going to be identifying whether or not it can perform multi step reasoning over long sequences with a high degree of reliability. So what I think Here is I'm 5050 on this model. I'm honestly not overly excited about this model, nor am I overly unimpressed by this model. I would say I'm more confused than anything because this model doesn't show anything too remarkable. But what I will say is that if you can use this model, I would try and test it for long horizon tasks to see how it would plan out different things and then compare that against some of the other frontier models. Because if that is what this model is for, then it should be actually good at that. But of course, this is just the test. We have no idea how the final model is going to be if this was just one aspect of the model at all. So it will be interesting to see how this entire thing develops. Let me know what you guys think about this in the comment section below, and hopefully you enjoyed this video and I'll see you guys in the next one.