China STUNS OpenAI With NEW Model That BEATS o1 (DeepSeek-R1-Lite)

So incredibly, OpenAI has got some serious competition coming out of China. Recently. There was information coming out of this company called DeepSeek AI and they announced that their DeepSeek R1 Lite preview is now live and apparently has 01 preview level performance on some of the most challenging benchmarks. This is something that literally took the industry by surprise because it was only two months ago that that we got the release of O1 in its limited forms such as O1 mini and of course O1 preview. And those models demonstrated far advanced reasoning capabilities, far more than anything we have currently available from any other companies. But of course China have decided to1up OpenAI by releasing their R1lite preview, which of course you can use for free. Now this is going to be something that is absolutely incredible because many would have suspected that it would take at least a year or two, at least potentially eight months to catch up to what OpenAI has. But apparently China has been able to do this within the span of two months. So if you're wondering who this company is, there isn't that much information about them publicly. But I do know that Deepseek AI was founded in 2023 and it's a Chinese company that is focused on advancing AGI and the organization specializes in developing cutting edge AI models that excel in tasks such as coding, mathematics, reasoning and and natural language processing. And they've actually made some previous models including deep seq version 2.5, deepseek coder and those ones were actually achieving a top performance on major AI benchmarks. Now if we get into the recent benchmarks, you can see that when we actually take a look at what we're seeing here, it's truly incredible. With as to how much progress they've made in such a short time, we can literally see that the Deepseek R1 Lite Preview manages to surpass 01 Preview in several certain categories. If we look at the AIME benchmark, which is a math benchmark, you can see that Deepseek R1 Lite Preview surpasses O1 Preview at a 52.4 benchmark compared to O1 Preview's 44.6. We can also see on this Math 500 benchmark it gets 91.6 and 01 Preview gets 85.5. On the GPQA it doesn't match up to 01 Previews, but on Codeforces it surpasses. And on the other two OpenAI 01 Preview manages to excel. Now this is all incredible, but I do think later on I'm Going to talk about some nuance to this because I don't know how much compute they apply to each individual problem. So I do think there's potentially ways for them to get better results. But it will be interesting to see how all of that works. And if you wanted to see or screenshot something, this is a screenshot of, of all of the current benchmarks, like the most widely used ones. And literally on the math benchmark, you can see that Deepseek R1 Lite preview gets state of the art, which is 91.6, surpassing 01 preview's 85.5. And I think it's really incredible to see those bars. If you see the dark blue and the light green one, we can literally see that those two managed to surpass most of these other models in, in basically every category by a large jump, which is really, really surprising when we start to take a look at how this new method of test time compute is going to revolutionize AI. So now this is probably the graph that everybody wants to see. This is how these models scale with Test Time compute. Now, if you're unfamiliar with the test time compute paradigm, the long story short is that the more you allow a model to think think, or the more compute you allocate to certain problems, the better these models get at being able to solve those specific problems. Which is why they call it Test Time Compute. And which is why right now there is this entirely new paradigm where you essentially have a different number of thought tokens per problem and that allows you to continually scaling. And the craziest thing about this entire scaling law seems like there doesn't seem to be any end to the results that you get from this. So far. The more thought tokens per problem that we add, the better these models get at responding to extraordinarily difficult prompts. Now, the reason I said this one is going to be particularly interesting is because if we actually take a look at this purple bar, which is the O1 preview, which is from OpenAI, we do see that the accuracy on the AME benchmark with Test Time Compute scaling, it stays consistent. But of course, right now I don't think there's a way for you to change the amount of thought tokens per problem. Like, you can't say, think about this problem hard. So right now it's just there. But I will be intrigued to see how the accuracy does increase with OpenAI's models. Now, when we actually look at this chart, we actually break things down. I'm just going to just break it down for you in the simplest way possible. The first thing that I do notice is of course the.01 preview level, and that is of course the purple line, the dashed line going through the middle. Now, this line basically shows where OpenAI are on the AMY benchmark. I'm just going to call it the AMY benchmark, which is just a math benchmark. And I think the only reason that the accuracy isn't increasing here is, is because when you actually use these O1 models or these thinking models, you don't actually get to decide how many tokens it applies to each problem. It just automatically applies a certain amount, which basically means that for some questions it's not going to think about it at all, and for other questions, it will decide to think about it quite a lot. So this benchmark that you're seeing on this graph right here, you have to understand that it's pretty different because we're seeing an increase in number of tokens per problem for these other models. So when we actually look at the R1 Lite, we can see that this model is actually really good when it comes to this difficult benchmark. And what you can see in the blue, because at first this confused me, but then I just looked into it and it makes complete sense, is that we have two methods of actually breaking down these benchmarks. So you can see we've got two different ones, blue and red. The red is Pass at 1, which is basically where you just ask the model question and it's just passed at one try. And then of course, you get majority voting, which is where you have, I think, around several. I'm not entirely sure how many numbers you have, but you have several, several responses. So with the pass at one, that's just basically where we're giving the AI one chance. And then of course, with the majority voting, this is basically where you get multiple responses from the model. And then based on those multiple responses, you pick the one that frequently shows up as the winner. So that's why you can see that with majority voting and a test time compute, we get this extraordinary graph that just continues to go up with the amount of tokens per problem, which shows us another incredible scaling law with this majority voting, which is basically pretty incredible, but all of this stuff is truly insane because it shows us that on the back end, when we're really trying to work through a lot of hard problems, we're actually going to have to burn through a stunning amount of tokens in order to get to the correct response. But overall, what we can see here is a Pretty clear trend. The company Deepseek AI have truly managed to catch up to where O1 preview is. And considering the fact that they're using things like majority voting on these benchmarks, it does show us that there is no limit currently from what we can see. I mean, the graph continues to go up. It seems like there's no limit to how these scaling laws will end. And some people have even predicted that companies right now like Deepseek are in the GPT2 era, which means the next iteration of these models is quite likely to be completely incredible in terms of their performance. Because I'm wondering, what if they manage to bake majority voting into the model so it intrinsically generates time, tons and tons of responses and then of course it picks the best one. You could see that if that was something that was baked into the model, then of course that would push the benchmarks even further. So this is something that I think is absolutely game changing because this truly shows us that we have entered a new paradigm. And the fact of the matter is, is that OpenAI, yes, they've got a lead, but it doesn't seem like they've got one for long. Now there also is something that is truly fascinating that I wanted to talk about. One of the things that OpenAI actually did, which a lot of people had, I guess you could say, a disagreement with, was the fact that they hid the chains of thought. So essentially, chains of thought are basically where if you're thinking through a problem, the user is essentially going to be able to see what the AI model is thinking about. So for example, if you say, okay, how would I make a cake from scratch? Think step by step. The model is going to be thinking, okay, what kind of cake do I need to ask to use? What kind of cake? Let's think about this. Maybe I should pick a common cake. I'm going to pick a lemon cake. All of those thoughts that the model is is thinking up before it manages to respond. OpenAI actually chose for their thinking model to completely hide that. So if you use 01 Preview or 01 Mini, you actually cannot see what the model is thinking. You can see a very, very short summary, but you can't actually see what the model is thinking. Now, of course they say that this is something that they do not want users to be able to do. They say we also do not want to make an unaligned chain of thought to directly visible to users. Now, I do think that they probably just did this for competitive advantages. I mean, they ideally don't want people to see how their models are thinking in order to get really good responses on the benchmarks. But the craziest thing about this, and why I brought this up, is the fact that Deepseek, they've actually shown the chains of thought. And the chains of thought are absolutely incredible for these models. And it kind of makes me think maybe just a little bit, that they might be a little bit conscious. But that's for another video. But let me show you guys what I'm talking about. So I saw this on Reddit, which is basically says Deepseek R1 Lite is surprised by the third R in Strawberry. So you can see here someone asked the very famous AI question, which is how many Rs there are in Strawberry. And you can see that the model thinks to itself saying, okay, this is not an R, this is not an R. Wait, there's a third R. And you can see that the model is getting surprised by this letter in the word Strawberry. You can see right here at the top, the model says, wait a minute, did I just count three Rs? Let me make sure I didn't miss anything. Strawberry. And you can see it literally says, yeah, that's three Rs in total. But. But I feel like that's too many for such a short word. Maybe I'm miscounting or misreading the word, which is a strange thing to say when you think about it. I feel like this da da da da da when you're talking about an AI model, which I'm not getting into the consciousness debate, trust me on that one. But I still think that this internal chain of thought is just pretty incredible to read. Now we also got more comments from Reddit stating that this preview thinks for over six minutes and even GPT4O and Claude3 couldn't solve their specific problem, which is pretty remarkable because Thinking for around 372 seconds is a whopping amount of tokens and a whopping amount of compute for something that is a very fascinating problem. Now, I also got this example from Twitter. I'll leave a link to this one, but you can see this one right here was pretty interesting. It said write a grammatically correct sentence without using any letters more than once. And it's thought for three seconds and it was able to get a repeated sentence and it was able to get the correct sentence. And then when you look right here, you can see that 01 actually didn't get this one right. It said Glib Jocks, Quiznimps, Devex, Dwarf. And there were two O's, jocks and two, which is of course not great. So overall, if you can actually come onto the website, you'll be able to see this entire internal reasoning and you'll be able to just click this and you'll be able to see exactly what the model is thinking and how it thinks through your prompt. Now. Now, the reason I think this is one of the most powerful things and is because this actually gives us the insight to how the model thinks. And the reason this is just absolutely insane, and I've recently done this, is if we can actually identify where these models are going wrong with their thinking patterns on a certain problem, we can actually craft prompts that are even better, that allow them to solve the issue and thus get even better at prompt engineering. So the user example that I'm showing here is a very popular prompt. This one says, assuming the laws of physics on earth, a small marble is placed into a normal cup and the cup is placed upside down on a table. Someone then takes the cup and places it inside a microwave. Where is the ball? Now explain your reasoning step by step. So basically this is a question that tests to see if these models have a world model and tests to see if it understands that if you turn a cup upside down with a marble on it and you pick that cup up, the marble essentially remains on the table. But firstly, when I actually spoke to this model, it actually said that the marble is still inside the cup, which is now placed inside the microwave. And interestingly enough, I thought maybe this is just about the prompt. And I would argue that it is actually about the prompt and not about how smart the model is. And the reason I was able to realize that is because when I looked through the chains of thought during the prompt, I could see that for some reason the model was thinking that the way how I was picking up the cup would actually carry the marble. And all I did was I made some slight iterations to my prompt and you can see that it says that the marble is on the table outside the microwave. So I, I think with this, what the very best thing you guys can do with is that if your model is currently getting distracted or confused about certain things, you can look exactly at what's going on inside, look at where it's getting tripped up, reorganize your prompt, and then you can use that for other similar problems, which I think has remarkable use cases. So with that being said, let me know if you guys are going to try out this model. I think it's absolutely crazy that Deep seek have been able to catch up to OpenAI01 that quickly. And I do think that this is probably going to actually force OpenAI to, to move even quicker because they definitely did not think that companies would be releasing these models that soon. It was probably well known that the next models that were going to be coming were probably like Claude 3 opus and of course Google's new model. But now with this other Chinese company that are hot on the heels of OpenAI, I think people are really, really going to start accelerating in terms of the aggressiveness.