Researchers STUNNED As A.I Improves ITSELF Towards Superintelligence (BEATS o1)

So this is absolutely incredible. Microsoft have released a research paper that actually talks about an LLM that can truly self improve. I know it might sound like sci fi nonsense but trust me, this is absolutely incredible. So take a look at this research paper called r math. Small LLMs can master math reasoning with self evolved deep thinking. And this is pretty crazy because essentially the model is able to use its own thinking to make itself even smarter. We can see that it says we present R Star Math to demonstrate that small language models, SLMS for short, can rival or even surpass the math reasoning capability of OpenAI01. And this is guys, without distillation from superior models. Like that sentence is just crazy. And if you don't know what model distillation is, Model distillation is a process where you have a larger teacher model and essentially transfers the knowledge from a larger model into a smaller model. And how that works is essentially the large model is pretty much called the teacher model. And the teacher model's knowledge is essentially distilled down into the smaller model and the student model is essentially fine tuned to perform similarly to the teacher model. So you know how when we had O1 preview and we had the actual O1, this was essentially what we had. We had the actual O1 and then we had the O1 preview which was essentially a similarly fine tuned version that just performed a bit less in terms of the actual capabilities of the teacher model. Now what they're stating in this paper, which is why it's so crazy, is that they can actually beat these large language models without that distillation. So it's absolutely crazy. And they say that, you know, R Star Math achieves this by exercising deep thinking through Monte Carlo tree search which is basically a form of an AI searching through this ranges of possibilities. And it's pretty crazy. Now it gets even crazier because when we take a look at the initial benchmarks, we can see that on the math benchmark it improves QEN 2.5 math 7B which is a 7 billion parameter large language model from 58.8% to 90% and 53 mini 8B from 41.4% to 86.4% surpassing 0.1preview by 4.5% and 0.9% on the USA Math Olympiad. Now if you don't know why this is such a big deal, one of the largest problems that people have had is essentially where people have thought that these small large language models, well they're small language models, have actually just been trained on all the training data in order to get those high marks on the benchmarks. But the crazy thing about this is that this model improves itself in order to get to this level. So it isn't trained on all of that stuff, it's able to improve itself. Now this is the craziest thing because this is where we have the diagram that provides an overview of the R Star math system, specifically focusing on the Monte Carlo tree search. We can see that over here and that this is the self Evolution framework. So this entire thing is how the model is able to think in a way that allows it to bootstrap itself to greater intelligence. So on the left you can see we've got this Monte Carlo tree search thinking. So this is where the system, you can see it explores these multiple paths. Think of it like a person trying to think about the consequences of their consequences or just thinking like a decision tree. Now, the small large language model acts as a policy to generate reasoning steps for the solution. And, and then we have a process preference model which is basically something that just verifies that each step is correct. And I'll dive into more about this later because it's actually pretty cool what they've done. Now, each one of these nodes that you can see here actually represent a step in solving the problem. So you can see right here, this one is 0.7, this one's 0.5 or whatever. But these are the steps that the model is going to take. And you can see right here that the ones that are incorrect are granted low values and the ones that are correct are granted higher values. And this is where the system essentially assigns Q values to each step based on the contribution to the final solution. And basically, steps leading to correct answers get higher Q values and steps leading to wrong answers get lower Q values. And essentially how they do this is that they filter out these steps so that only the high quality steps, the ones in green, are essentially retained to construct the final solution trajectory. And this basically filters out and makes sure that the AI trains on the best reasoning paths. Now what's crazy about all of this? And this is where you get this self evolution framework, because while Monte Carlo tree search is good, you do need to add something on top of that in order to get a language model to be able to improve itself. So this is where we get this four step process of where the system improves itself, enhancing both the policy, the small large language model, and the reward model, the ppm. So essentially you can see that there are four steps here, and this is where you guys want to pay attention. Two, it first starts with terminal guided Monte Carlo tree search, which you guys can see right here. Then you can see at round two it moves to introduce PPMR2, which scores the results from the previous steps more effectively. And then the data generated in this round further actually improves this model. So you can see that the model right here is R1. Then it changes and gets upgraded to SLMR r2 small large language model. R1 then gets upgraded to Small Large Language model. I don't know why I keep saying small Large Language model. It's Small language model, but it gets upgraded to R2. Then at round three, the policies manage to leverage the reward model to directly predict the Q values during the search process. And then it generates higher quality solutions and training data. Then this trains the next iteration of the model, Small Language model three. And you can see now we get to version three of the model. And then at round four, this is where the final model emerges and we get the continuation of the reward model with an even stronger policy. And this is where it manages to achieve state of the art performance. And this is absolutely incredible because the model starts here, not knowing that much, and then eventually it manages to iteratively improve itself to becoming state of the art through this process. And it's just absolutely incredible because it's generating all this synthetic data and it's able to do that with this process. Now when we actually take a look at how these benchmarks are, you can see at how the model is able to continuously improve its math reasoning capabilities through self evolved deep thinking. So you can see that by round two, the model actually surpasses GPT4.0. So we can already see what the base model is capable of achieving. You can see that the base model is capable of achieving 58.8 on math. You can see it doesn't get any on Amy, none on this other benchmark. And you can see that it just doesn't do that. Well, then of course it begins the improvement process outlined in the previous slide. And then you can see it jumps up to 75.2, then 10, then 50, then 35. And you can see it improves a decent amount. Then you can see at round two on the improvements after round one, we can see that the model is already surpassing GPT4. At round three it gets even better and around four it becomes state of the art. And this is where the model can even surpass 01. And remember guys, this model is a 7 billion parameter based model and it's surpassing state of the art models. Now of course, this is one specific benchmark. This is specifically focused on math. But let me actually think about how crazy that is. We can start to imagine how incredibly smart these models are going to get if a 7 billion parameter base model can even outperform O1 or GPT4O. Imagine a mixture of experts of these models that are incredible and are able to improve themselves. What kind of model are we going to be getting then? Now I wanted to talk about how does this model actually improve itself? The basics of this is that the model generates solutions, then it evaluates the solutions, and then it retrains on the best solutions, and then it repeats. And that entire feedback loop that we have here is something that, you know, continuously refines its reasoning abilities. And if you might be thinking, okay, but what about the unsolved problems? So for problems it cannot solve, the model essentially uses multiple reasoning attempts, which are essentially additional rollouts or new random seeds until it finds a solution. And this basically ensures that the model is always learning from its failures. And I don't think you guys understand how crazy this is. After the initial round, it generates its own training data. And this is a major shift away from traditional training methods which rely heavily on distillation from these larger teacher models. Now, this approach is incredibly powerful because self generated training data actually eliminates the need for manual labeling or large datasets. And in this paper they actually talk about the fact that doing that is actually very time consuming. This is also cost effective and efficient, especially for specialized tasks like math reasoning. And the model isn't limited to its own initial training data, which means this can improve over time. Now remember how in another paper, maybe you watch the channel, maybe you don't, but they spoke about how when predicting what OpenAI's 01 is based on, they spoke about, you know, outcome based reward models, which are essentially where you just look at whether or not the outcome is right or not, and how they moved to process based reward models. And this is where basically, let's say you were doing a math question and you were outlining it step by step and you were writing down every single step, you basically get rewarded for every single step that is correct. But even if the final output isn't correct, you still get some rewards. So for this one, even if there are multiple steps that are correct, it's just all wrong. We don't care, we're going to throw it away. But if the first three steps are correct and the last two aren't correct, you're awarded for the first three correct steps. Now what I Found to be like super, super interesting for this is that they actually changed the reward model. So they used a process preference model. And they said that we introduce a novel training method that trains a process preference model by constructing step level positive negative preference pairs. Let me just explain this for you guys. So basically they've changed what they're doing here. They're using essentially a hybrid approach of process reward modeling and basically the P RPM method that this is process preference model. It trains a model to guide the reasoning process by comparing these steps. And this avoids assigning precise scores to the steps, making it more robust and reliable. And this is the core innovation in the paper that's used to drive the Monte Carlo tree search process. So essentially overall, when we look at the ppm, the process preference model, this is the backbone of R star math, especially in the later rounds where it outperforms traditional process reward model approaching. And they essentially use the process reward model initially for bootstrapping the model, but the transition to more effective process preference modeling once the system matures. And this is a hybrid approach that actually balances the two benefits. So now let's look at the axes. So the bottom here on the X axis we can see this shows us the sampled solutions. So we can see as the sampled solutions increase, the accuracy of this increases. And this is something that we know, but this basically shows you how many solutions are sampled. And on these graphs that we can also see is we see the accuracy on the Y axis and we can also see the different models. So we can see that the green model, the R star math, we can see the performance of this model. We can then see the Qin 2 models, the best of N, which is a different process. We can then also see the O1 mini accuracy and the O1 preview on them. And we can see them across these all different benchmarks. Clearly, 01 preview wasn't tested on the Olympiad bench or the college math, but O1 mini, we can see the levels where it was in terms of the accuracy. And what's crazy is that as these sampled solutions, as we decide to sample more, we can see that the R Star math model manages to get smarter in its responses. And remember, this is a 7 billion parameter model that is going toe to toe with these much larger models. And this is something that does actually appear to be true for all models. We can see that for all models they seem to get smarter on the benchmarks as they continue to go up. Of course there is a first jump. We can see that from one to four solutions there is a Decent jump. But of course as you manage to sample more solutions, it does seem like the gains, although they do increase, it does seem like they are less and less marginal. But what is interesting is that with R Star Math we can see that even for less sampled solutions, it manages to perform much more effectively than a lot of these models. And it's really surprising that with just sampling more solutions and a different way of processing the information, the model manages to get to higher levels of reasoning. And something that I am just looking at once again in further detail is that we can see that this model right here, the R Star Math model, is actually performing better than this 72B model. Which just goes to show that this method is definitely really effective. And it's giving me the information that by using Monte Carlo tree search and the process preference model, it's showing us that this is something that is really effective that maybe we need to explore with large models. And I think, you know, now that I'm starting to read these research papers in more detail, we're starting to see that maybe the claims of superintelligence aren't that far fetched. And essentially when we look at this, the ability to achieve such a high accuracy with fewer sampled solutions makes R Star Math essentially really cost effective and scalable, particularly for tasks requiring mathematical reasoning. Now another interesting thing that I found in the paper was that it also generated its own data as we spoke about. But the crazy thing about this was that its own generated data was better than GPT4. We can see here that it says even randomly sampled code augmented chain of thought solutions from our small language model yields comparable or better performance than GPT4 synthesized numath and metamath datasets. This indicates that our policy SLMs after rounds of self evolution can generate high quality math solutions. And it says these results demonstrate the huge potential of our method to self generate higher quality reasoning data without relying on advanced LLM distillation. And this is basically incredible. This statement refers to the randomly selected reasoning paths generated by the model during the training. And even without selecting the absolute best solutions, all of the random samples, these ones are much more high quality. And that's pretty crazy because those Numina Math are essentially synthesized by GPT4, which is a much larger and powerful model. And these data sets, Numina Math and Metamath, are considered high quality data sets. So when we talk about them surpassing them, that actually demonstrates the strength of R Stars Math and the self evolution process. And the quality of these reasoning solutions isn't just luck. This is the result of self evolution. And through the iterative rounds that we talked about, these four rounds, the SLM becomes skilled enough to generate solutions that rival those produced by an advanced model like GPT4. And this is absolutely crazy when we are now breaking the dependency on large language models and we have to think about this, let's say we had a model that was much bigger and we were able to generate methods that were smarter than smarter models from the future. I know that sounds like it doesn't make sense, but when we look at how we're taking a 7B model and making it generate things that are smarter than GPT4, which is a 1.8 trillion parameter model, we can think, okay, what about if we have a much larger model and we focusing on making that one using these kinds of methods to improve itself iteratively over time, when we actually take a look at what that means for the future? That is going to be crazy. Now something crazy about this paper as well was that they spoke about how there were emergent capabilities. And emergent capabilities in AI are essentially capabilities that arise without prediction. So these capabilities arise without them training them, without them essentially specifically wanting those capabilities, or without them thinking that the capability may arise. And we can see here that they put in chapter five in the findings and discussions that there is the emergence of intrinsic self reflection capability. It says a key breakthrough in OpenAI's Zero1 is its intrinsic self reflection capability. When the model makes an error, it recognizes the mistake and can self correct with an answer. Yet it has consistently been found to be largely ineffective in open sourced LLMs, which is of course the models that they're using right now, which is the Phi series and the Quinn series. And it says the community has actively explored various approaches, including self correction, which is where they self correct, and of course self reflection to explicitly train or prompt LLMs to develop such capability. So they're talking about where people previously used other models to try and get them to actually, you know, have that as their standard way of thinking. But in their experiments, they unexpectedly observe that our Monte Carlo tree search driven deep thinking exhibits self reflection during problem solving, as shown in Figure 4, which I'll show you guys in a second. The model initially formalizes an equation using this in the first three steps, which would lead to an incorrect answer. But in the fourth step, the model recognizes the low quality of its earlier steps and refrains from continuing along the initial problem solving path. Instead, it backtracks and resolved the problem using a new, simpler approach, ultimately arriving at the correct answer. And the crazy thing about this is that we can actually see this right here. So we can see that this is where it's going to solve the solution. We can see it manages to get it wrong. And then this is where it has its intrinsic self reflection, thinking outside the box to find an easier solution to the problem. And we can see these scores as the image progresses. We can see the PPM scores are negative here. And as we go down here we can see the PPM scores are increasing in value. And remember they said notably no self reflection training data or prompt was included, suggesting that advanced System 2 reasoning can foster intrinsic self reflection. Which is, I mean, I want to say now I'm starting to understand where the AI hype is leading. Because if they can get a small language model to achieve the math reasoning capabilities of OpenAI's 01 through a thinking process that essentially is search and test time compute, and through four rounds of evaluations and through multiple sampled solutions, they can actually iteratively improve the model. I mean, what happens when we start to improve models that are much larger and we have much larger compute budgets? It's quite likely that we are going to get models that are so smart that we won't even know what to do with them. So it's going to be pretty crazy when we're thinking about all of the kind of solutions that these models are going to be able to come up with. And I'm starting to understand why. Now in a few interviews, certain people at Microsoft were talking about the fact that we could get fully self improving AI in, in 2030. Take a look here.

That's a narrow version of AI when we use it for a specific purpose. Right. A more general AI is going to be one where it has things like recursive self improvement. It could edit its own code in order to get better, it could self improve or like it would have autonomy. It could act independently of your direct command essentially or you give it a very general command, you know, and it goes off and does all sort of sub actions that are super complicated, like you know, maybe even invent a new product and create a website for it and then set up a dropship for it and then, you know, go and market it and take all the income and then do the accounts and so on. I mean I think that's kind of plausible in say three to five years before 2030. I think we'll definitely have that and might well be much, much sooner.

And remember it was Eric Schmidt who essentially said that look, we are going to get self Improving AI in the future. So it makes sense that once we do, we need to be very careful and we need to sort of unplug those models. Now what he did also mention was the fact that that model is going to be a model that is recursively self improving, meaning that the model is able to improve itself in a way that is I would say kind of stateless. So it's able to adaptably improve itself over time. Which means that after completely training the model, the model is just going to be able to continually improve itself again and again and again overall, meaning adding new tools and abilities, which is just completely a different kind of way. Because that is the kind of intelligence that could ruin humanity. And that's why people are concerned about this. Because let's do a theoretical experiment. Let's say the model was able to continually improve itself. Let's say it was able to go out, use the tools, train on other pieces of data, collect things from the Internet, upload itself to humanoid robots or whatever, do AI research. I mean when we think about all of that and it has access to a large amount of compute, we can start to realize how these things can self improve beyond our current reasoning.

So we think we're okay now and we're worried about the future. And the, the point at which you really want to get worried is called recursive self improvement. So recursive self improvement means go learn everything, start now and don't stop until you know everything. And this could allow this recursive self improvement could eventually allow self invocation of things and imagine a self a recursive self improvement system which gets access to weapons so you can imagine or doing things in biology that we cannot currently understand. So there is a threshold. Now my standard joke about that is that when that thing starts learning on its own, do you know what we're going to do? We're going to unplug it. Because you can't have these things running randomly around, if you will, in the information space and not understanding at all what they're doing.

I think that when we actually look at this and the conclusion here says that you know R Star math. A self evolved system. Two deep thinking approach that significantly boosts the math reasoning capabilities of small LLMs achieving state of the art OpenAI01 level performance. Our approach demonstrates that SLMs can self generate high quality training data for frontier level math reasoning. And this is absolutely crazy. Extensive experiments across four different sized SLMs and challenging Benthmarks demonstrate the superiority of R Star math with achieving leading results while outperforming existing math reasoning, LLMs and best of n baselines. And we also reveal the key findings, including the emergence of self reflection and the effectiveness of the PPM in identifying critical intermediate steps such as theorem application steps. Finally, Ask Math can achieve further improvements by collecting more challenging problems and we leave this as future work. When we actually take a look at what we just read here, I mean it's absolutely crazy. The test time compute paradigm is clearly real. Having a model sample more solutions and search through those solutions is something that clearly is the next paradigm. OpenAI has spoken about this in so much detail and when we've actually applied this to a 7 billion parameter model and used process preference modeling to identify the key steps, we can see that this results in an incredible level of model capability. It was generating synthetic data that was better than the data that was currently available from GPT4O. And crazily enough this is something that outperforms model distillation. So yes, this is something that is in one area I would say that is the only caveat is that this is only in math. But of course they actually do talk about the generalization discussion. So they say that, you know, R StarMap offers a general methodology for improving reasoning applicable to various domains, thus can generalize to more challenging tasks such as theorem.

I thought it would be much harder.

Focus on word problems due to data set limitations and nonetheless our star Math demonstrates the potential mathematical state as shown.

In a it's successfully seriously about an.

Ellipse level problem involving the mathematical theorem providing a step by step correct proof through its deep reasoning process. Now this bit right here is just pretty crazy because a mathematical theorem step by step actually requires deep reasoning, logical consistency and advanced understanding. And most AI system will, you know, they'd actually struggle with such a task because that require logical deductions and not just memorization. They involve multiple layers of reasoning to connect abstract concepts. And the fact that R Star Math can handle a proof of an Olympiad level problem like Fernand's little theorem suggests that it isn't limited to solving straightforward computational problems, but it can construct complex proofs reflecting reasoning akin to how humans approach mathematics at a high level. Now traditionally, you know, this is, you know, AI systems, you know, they've really excelled at solving structural problems like word problems with clear answer. But this model, when we actually take a look at things, it's actually taking it a step further by showing it that, you know, it can handle formal proofs that involve logical rigor, abstract reasoning and a need to justify every Step mathematically. So this transition is somewhat of a paradigm shift. And you know, it's a paradigm shift in terms of what these smaller models can achieve, rivaling the capabilities of larger, more resource intensive models. And when we look at the second part right here, it talks about how this model is able to generalize to other domains such as code and common sense reasoning. And it says, notably, synthesizing step by step verified training trajectories for general reasoning requires a mechanism to provide feedback on whether a given trajectory reaches the dyed output at the end of the Monte Carlo tree search rollout. For instance, in code reasoning this could involve designing extensive cases in general reasoning and feedback could be obtained through human labeling or mutual verification with another LLM. So this is pretty impressive because code reasoning requires understanding syntax, logic and problem solving all at once. And if this system is optimized for math and it's able to be optimized for other domains like programming, this is going to be a leap in versatility. And they're basically stating that if this is able to generalize the code weaseling, it could be able to write a function or script step by step, then run test cases to check if the function works correctly and improve its reasoning based on the test results. Just like it validates the math solutions using Python execution. I do have to say though, that would be particularly difficult because, you know, one thing that was being discussed on Twitter the other day was that, you know, it's quite hard to get correct answers for questions that don't really have a definitive answer. But whereas math, where you have a definitive correct answer, it's quite easily to get verifiers for that. And you know, common sense problems are a bit more open ended. But at the same time this is incredible research that shows us that we are clearly every single day edging closer towards AGI. But at the end of the day I still find that this is completely crazy because when we think about this, when we look at the overview of the system, it is math that underpins many other things. And once you're able to truly, truly solve math, which we know that OpenAI's O3 model was able to do really well at math, yet 25% on pretty much an unseen benchmark. Although that training round cost like, I guess you could say the inference that time was around a million dollars, I think we can understand that the trajectory of these models now is going to be something that's absolutely insane, and that's why people are so bullish on artificial superintelligence, is because on early research papers, these researchers were stunned because one of the things that they realized was that these graphs, like these numbers, they weren't going down, they just kept going up as long as you managed to sample more solutions. So over time, as long as we managed to sample more solutions and get more effective at learning how to identify the correct reasoning steps, I don't think that there's going to be a problem that these models aren't going to be able to be solved. And when we understand that paradigm, we can understand that maybe those talking about artificial superintelligence aren't that far off after all.

Both Demis and I believe we are going to get super intelligence. It's not hype and it's not designed to distract from other problems with AI. It's what we have believed for a long time now. I thought it would be much further away, but the speed of recent developments means I think it's going to be quite soon. I think between five and 20 years. I think Demis thinks in about 10 years we'll get something like super intelligence and we have to worry seriously about how we stay in control then.