Open AI's Q* Is BACK! - Was AGI Just Solved?

So if you thought Q Star was dead or that we weren't going to get any more updates, think again, because recently we actually got a very small inkling that Q Star might just be true. Now, Q Star did actually captivate the AI community for several reasons, because there were so many things going on. But recently I saw this tweet on my timeline, which was a reference to a research paper that actually showed how tiny large language models are actually good at maths as a frontier model. So by using the same techniques that Google actually used to solve AlphaGo, which was Monte Carlo tree search, and of course back propagation, llama 8 billion parameters, llama 3 gets 96.7% on the math benchmark GSM8K, and that is better than GPT4, Claude and Gemini with 200 times less parameters. So this is a pretty, pretty crazy revelation because it was something that I didn't see coming just yet, because the entire framework of applying search to LLMs is something that I wouldn't say it's still in its early stage, but it isn't something that we didn't really think about initially when we went from GPT 3.5 to GPT 4. And there was a lot of talk about this being explored. And clearly we're now seeing that this is being explored and the results of that are, are truly, truly impressive. Now I am going to dive into a little bit of this research paper and if you don't remember, Q Star, Q Star was basically some kind of, you know, crazy thing, because it was around the time that Sam Altman got fired, there was this information article that, you know, spoke about how OpenAI made a breakthrough before Sam Altman got fired, which was stoking excitement and concern. And of course, the crazy thing about Q Star that you need to know is that Q Star was able to solve new math problems that it hadn't seen before, which are an important technical milestone. And apparently a demo of the model circulated within OpenAI in the weeks, and the pace of development alarms some researchers focused on safety. Now, what is crazy about all of this is that the team who are working on this, Sutskova, were actually working on ways to allow LLMs like GPT4 to solve tasks that involved reasoning, like math or science problems. And in 2021, they actually launched a secret project called GPT0, which was a nod to DeepMind's AlphaZero program that could play chess, go and shogi. So they were already kind of working on this in those early, early days where they launched a project called GPT0. So this is not something that is from a fundamental like thought point of view, something that people haven't really thought about before. This is something that you can see some of the top researchers have have thought before and they initially hypothesized that by giving large language models more time and more computing power to generate responses to questions, they could allow them to develop new academic breakthroughs. So of course that is pretty crazy because what we see in this paper here is that they actually did this. Basically they actually used GPT4 accessing GPT4 level mathematical Olympiad solutions via Monte Carlo tree search, which with self refine with llama3.8 billion parameters Technical reports you can see right here, they state that Monte Carlo self refined algorithm represents an integration of Monte Carlo tree search with large language models, abstracting the iterative refinement process of mathematical problem solutions into search tree structure. Nodes on this tree represent different versions of answers, while edges denote attempts at improvement. This algorithm's operational workflow adheres to the Monte Carlo tree search algorithm's general patterns. The basically saying that they're basically doing the same thing that they did with AlphaGo, they're using that same kind of structure in LLMs. And the craziest thing about this was that Andrej Karpathy actually spoke about the fact that if you actually use Monte Carlo tree search, which is something that they used in AlphaGo and basically how this kind of works. Because I know I'm saying this and I know that a lot of people might not be familiar with all of the news from the past couple of months, but basically what they're doing is they're basically having an AI system search over all the possible configurations before making that move. And then once they can make that move, they can then of course on the next move, search again of all the possible board configurations and then go ahead and make the move. So that is essentially how that works. And it's something that Andrej Karpathy describes in his one hour talk about language models where he basically says that this is something that we kind of need for future models if we want things to improve. And it's pretty interesting that now we're seeing a paper that, you know, is so small in its size, be able to, you know, if we look at like the benchmarks and stuff, you can see that on the GSM 8K, on the eight rollouts, like on the eight rollouts that it did, it achieved 96.66%. Call it 97. And if you actually do check some of the frontier models it literally surpasses them, not by a large amount, but you got to remember guys, this is 8 billion parameters compared to a model that is 1.8 trillion parameters. And of course Gemini Ultras, I'm not even sure how many parameters that is not publicly available, but I know it is a pretty, pretty large language model. So basically this is Andre Karpathy status. I think this is his one hour talk is definitely fascinating. I really do believe that it's something that everyone should watch because gives a lot of insight into what's going to be coming in the future. But take a look at this because it's kind of important.

A lot of people are broadly inspired by what happened with AlphaGo. So in AlphaGo, this was a go playing program developed by DeepMind and AlphaGo actually had two major stages. The first release of it did. In the first stage you learned by imitating human expert players. So you take lots of games that were played by humans, you kind of like just filter to the games played by really good humans and, and you learn by imitation. You're getting the neural network to just imitate really good players. And this works and this gives you a pretty good go playing program, but it can't surpass human. It's only as good as the best human that gives you the training data. So DeepMind figured out a way to actually surpass humans and the way this was done is by self improvement. Now in the case of Go, this is a simple closed sandbox environment. You have a game and you can play lots of games in the sandbox and you can have a very simple reward function which is just winning the game. So you can query this reward function that tells you if whatever you've done was good or bad. Did you win yes or no. This is something that is available, very cheap to evaluate and automatic. And so because of that you can play millions and millions of games and kind of perfect the system just based on the probability of winning. So there's no need to imitate, you can go beyond human and that's in fact what the system ended up doing. So here on the right we have the Elo rating and AlphaGo took 40 days in this case to overcome some of the best human players by self improvement. So I think a lot of people are kind of interested in what is the equivalent of this step number two for large language models because today we're only doing step one, we are imitating humans. As I mentioned, there are human labelers writing out these answers and we're imitating their responses and we can have very good human labelers. But fundamentally it would be hard to go above sort of human response accuracy if we only train on the humans. So that's the big question. What is the step two equivalent in the domain of open language modeling? And the main challenge here is that there's a lack of a reward criterion in the general case. So because we are in a space of language, everything is a lot more open and there's all these different types of tasks and fundamentally there's no like simple reward function you can access that just tells you if whatever you did, whatever you sampled was good or bad. There's no easy to evaluate fast criterion or reward function.

So yeah, that makes a lot of sense because in math, slash, language, it's actually, you know, I mean, I guess in math it's obviously a bit easier. But of course in language things are pretty much open to interpretation. Like how do you judge whether or not advice is good or bad? You know, it's pretty subjective, but within this, it's just like if you win a game, you win a game and then of course you can easily train on that. Like, of course that's an easy reward function to use. But the point here is, is that when we look at this, we can see that this kind of architecture where you have learning that's not just based on human inputs is truly different because it actually allows the system to become super intelligent. So the one that they actually built that was basically the biggest and most superhuman, actually didn't train on human data, which is pretty crazy. So we had this AI system that was literally able to search over multiple different moves and that that was a key factor in its success. So basically right here in the AlphaGo documentary, they actually talk about how, you know, AlphaGo was being able to search over, you know, 50 or 60 moves ahead. And that's how it was able to get, you know, this remarkable level of accuracy. 50 or 60 moves ahead. That's the maximum number of moves ahead that AlphaGo is looking. From the current game position, it's typically over 50. It's often over 60 in the games we see, often about 150. AlphaGo goes for the kill. We're 115 now, so we're getting to that later. So yeah, it's a pretty interesting documentary. I'm sure many of you guys have probably seen it. But I think this finding is definitely really intriguing because it does show us that a lot of the work and a lot of the leaks that we've seen in some of them are kind of gaining more validity because it shows us that when people were talking about how LLMs plus search could be a very, very fascinating thing to discover and of course it could lead to potentially maybe superhuman capabilities or maybe even the capabilities that just vastly exceed human capabilities. I mean, right now we're seeing that with this paper, the initial results are truly, truly surprising. Like a 7 billion parameter model outpacing GPT4O on the GSM8K is pretty, pretty impressive. Which shows us that, I mean, like people have said, like Leopold Aschenbrenner said, what kind of growth are we going to be experiencing over the next couple of years when people actually develop literal ways to just improve the models that we do have now with different prompting techniques and different ways to actually use the base models that we have now. So this is definitely something that is very, very fascinating. Combining LLMs with search is yielding a true, true expansion in terms of the capabilities. Now what's crazy about this as well, something that I think that most people did also miss was that combining LLMs with search is going to be a big thing in the future. But the one major thing that is actually stopping it is compute. Because basically AlphaGo was pretty, pretty compute intensive because you're searching over so many different methods. But essentially one thing that you need to know is that Google actually did an AlphaCo2 paper and I did a video on this, but it didn't get that many views compared to the Gemini news because the Gemini news was basically just stealing the spotlight. And the thing about that was the fact that a lot of people overlooked what AlphaCode 2 was because it was a very fascinating insight with as to what is going to come for the future. So basically AlphaCo2 and this relates back to this before because basically it also uses a search algorithm and a RE ranking mechanism. But this isn't Monte Carlo tree search. But the point is that when they combined language models and a bespoke search and RE ranking algorithm, they were able to form 85% better than competition participants, which is a huge, huge, huge improvement. And basically they essentially used a sampling mechanism that encouraged generating a wide diversity of code samples to search over the space of possible programs. And basically what they did was they built a model, like a large language model kind of thing. And basically they combine this with advanced search, you can see right here, and re ranking mechanism tailored for competitive programming. And this thing was really, really good at competitive programming, guys. Like this was really, really good. Okay, and how did it get really good? They combined it with search and they were able to get these, you know, really insane capabilities on coding, which is why, you know, I'm bringing this back to Q Star because it shows us that these capabilities, you know, when you combine them with search, it's. It seems that some very, very, very intriguing initial findings here. Now if we look at the AlphaCode search, you can see here as well that it says our sampling approach is close to that of AlphaCode. We generate up to a million code samples per problem using a randomized temperature parameter for each sample. To encourage diversity, we also randomize targeted metadata, including the prompt such as problem difficulty rating and its categorical tags. And it says massive sampling allows us to search the model distribution Thor and generate a large diversity of code samples, maximizing the likelihood of generating at least some correct samples. So overall, you can see here as well, like as well, this is what I wanted to show you guys is that despite AlphaCode 2's impressive results, a lot more remains to be done before we see systems that can reliably reach the performance of the best human coders. Our system requires a lot of trial and error. I wouldn't say they maybe solve the problem of programming, but the point is that like with search, they've had some very, very impressive results. The only problem here is that it's too costly to operate at scale, which means they can't scale this thing. And that's of course kind of like an issue if you of course do want to use this in your products or whatever. So of course there are some things to work on in terms of optimization and how you kind of get that down. But overall, when we look at this, you can see here that it's really, really incredible because you can see that, you know, they demonstrate a clear trend where increased rollouts correlate with higher success rates, highlighting the algorithm's potential to improve performance through iterative refinement. And they also say that these findings affirm the Monte Carlo self refined algorithm's robustness and its utility in tackling complex unseen mathematical problems. I am wondering if this is very similar to one of the pieces of the article that said Q that was able to solve math problems that I hadn't seen before. An important technical milestone, a demo of the model so circulated within OpenAI in recent weeks and the pace of development alarmed some researchers focused on AI safety. Now what's crazy about this as well is that around the time of Q Star, I do remember that they hired Noam Bran. And basically what he did was he spoke on the Lex Friedman podcast about how in order to make superhuman Systems, stuff like Monte Carlo tree search, and being able to search over multiple different things is really important. In superhuman AI system.

It was very heavily focused on search, looking many, many moves ahead, farther than any human could. And that was key for why it won. And then even with something like AlphaGo, I mean, AlphaGo is commonly hailed as a landmark achievement for neural nets, and it is. But there's also this huge component of search, Monte Carlo Tree search, to AlphaGo that was key, absolutely essential for the AI to be able to beat top humans. I think a good example of this is you look at the latest versions of alpha of AlphaGo, like it was called AlphaZero, and there's this metric called Elo rating where you can compare different humans and you can compare bots to humans. Now a top human player is around 3600 Elo, maybe a little bit higher. Now AlphaZero, the strongest version, is around 5200 Elo. But if you take out the search that's being done at test time. And by the way, what I mean by search is the planning ahead, the thinking of, like, oh, if I move my, if I place this stone here and then he does this, and then you look like five moves ahead and you see like, what the board state looks like. That's what I mean by search. If you take out the search that's done during the game, the Elo rating drops to around 3,000. So even today, what, seven years after AlphaGo, if you take out the Monte Carlo tree search that's being done at when playing against the human, the bots are not superhuman. Nobody has made a raw neural net that is superhuman in go.

So, yeah, I found that clip to be pretty fascinating. The entire interview is on Lex Friedman. But I think one of the tweets, and I'm sure I've referenced this before, but this is why this tweet is so important and this is why I included the clip, because I think it goes to show that this person working at OpenAI, Noam Brown, you know, he was tweeting about these kinds of things and he was stating, okay, that, you know, all these prior methods are specific to the game, but. But if we can discover a general version, the benefits could be huge. Yes, inference may be slower, like a thousand times slower, and of course be more costly, but what inference costs would we pay for a new cancer drug or proof of the Riemann Hypothesis? So basically what he's saying here is that, look, if we could get a system that could really, truly understand certain areas and truly, truly, truly truly give us answers that are worthwhile, even if the speed of those answers is like a thousand times slower and they cost maybe not a thousand times more, but like 500 times more. If those answers are things that fundamentally change our level of understanding on certain topics, then entire new paradigms are going to be built off the back of that. So I truly believe that this is a step in the right direction because many people in the AI community, even skeptics like Yann Lecun, have actually spoken about this. And I've even seen Gary Marcus comment even under a few of these tweets and state that this is some good stuff. So it seems that this might be an area of further exploration for OpenAI, although we haven't really seen OpenAI state anything just yet because as you know, research from OpenAI is pretty much closed off because they are a private company now. I do think, interestingly enough, Sam Altman may have actually hinted at this because you remember how, you know, Monte Carlo tree search, you're basically searching, you know, to be able to see what kind of, you know, solution you can get. But Sam Altman in a Bill Gates interview, in a very, very, very short clip, he actually says, you know, something about that along the lines that I'm going to play it once for you guys here. You know, if you, if you ask GPT4 most questions 10,000 times, one of.

Those 10,000 is probably pretty good, but it doesn't always know which one.

And you'd like to get the best response of 10,000 each time.

So that'll be that, that, that increase.

In reliability will be important. So one of the things I'm wondering is, since he said that, you know, if you ask GPT4 a question 10,000 times, one or two of them is going to be absolutely amazing. So what if, what if like, you know, this is what they're working on for future systems, maybe they're working this on for GPT6, GPT5, I don't know, maybe that's why they need all these data centers, because a large majority of what they're doing is probably going to be, you know, maybe search based so that they can truly get, you know, reasoning that gives you the best answer. Maybe it literally just generates a bunch of answers, you know, whichever kind of search algorithm they do use. But these kinds of ideas are definitely a step in the right direction because I think this is something that everyone is truly agreeing on. So so far what we have here is a truly, truly fascinating paper. You know, some of my might got cut out. But I do think that this entire topic is very fascinating. I do really, really wonder what OpenAI are going to come out with next. I do believe that, you know, if they've got someone like Noam Brown on the team, they've already had, you know, whatever breakthroughs they've had with Ilya Satskova. And we're starting to see slowly that, you know, a lot of these other, you know, labs are starting to catch up. So as I was rendering this video, I actually saw a tweet on my timeline that kind of gave me a decent amount of information on something that was going on inside the AI community. And it was something that I watched recently in an interview with Dwarka Patel. He has, honestly, some of the most insightful interviews with some of the brightest minds in the artificial intelligence space. But, yeah, it's pretty crazy. So, basically, all right, the news is this. Okay, so there was this. And this actually does link back to the original Q thing that we just saw. So just bear with me a second. Basically, right there is this arc AGI thing that basically they're stating that this is the new benchmark that actually tries to prove whether or not a system is AGI or not. And you cannot achieve AGI without surpassing this benchmark. And that's the only kind of benchmark that people are going to consider. I'll show you guys a short clip of that interview.

Now, one ARC puzzle. It looks kind of like an IQ test puzzle. You've got a number of demonstration input output pairs. So one pair is made of two grids. So one grid shows you an input, and the second grid shows you what you should produce as a response to that input. And you get a couple pairs like this to demonstrate the nature of the task, to demonstrate what you're supposed to do with your inp. And then you get a new test input, and your job is to produce the corresponding test output. So you look at the demonstration pairs, and from that you figure out what you're supposed to do and you show that you've understood it on this new test pair. And importantly, in order to the sort of, like, knowledge basis that you need in order to approach these challenges is you just need core knowledge. And core knowledge is basically the knowledge of what makes an object, basic counting, basic geometry, topology, symmetries, that sort of thing. So extremely basic knowledge. LLMs for sure, possess such knowledge. Any child possesses such knowledge. And what's really interesting is that each puzzle is new, so it's not something that you're going to find elsewhere. On the Internet, for instance. And that means that whether it's as a human or as a machine, every puzzle, you have to approach it from scratch, you have to actually reason your way through it. You cannot just fetch the response from your memory.

So the core knowledge. So now after that interview came out, okay, you can see that he tells his co worker Ryan, and within six days they've beat the state of the art on the Ark and are on the heels of average human performance. Which would mean that they're teetering on the edge of something that some would consider artificial general intelligence. He says, on a held out subset of the train set where humans get 85% accuracy, my solutions get 72% accuracy. So this is pretty crazy. But what's interesting is that he said, I started on this project a few days before the Dwarkesh Patel recorded the recent podcast with Chollei. This was inspired by Dwarkish talking to my coworker Buck about ARC AGI and being like, come on, surely you can do better than the current state of the art using LLMs. And basically right here, this is the main argument, okay, the main argument is that LLMs are just mimicking patterns. They aren't true AI systems. It's just not possible to get to AGI with LLMs and what we're doing. However, you can see right here there's a meme and you're seeing that someone's just saying, why don't we just draw more samples and then we can just get infinitely better. So that basically stating that we might be able to get to AGI by just providing more samples now. So for context, right here you can see that ARC AGI is a visual reasoning benchmark that requires guessing a rule from a few examples. And its creator, F. Chollet, claims that LL. And then this is the crazy thing, okay, so it says Ryan's approach involves carefully crafted few shot prompts that he uses to generate many possible Python programs to implement the transformations. He generates 5,000 guesses, selects the best ones using the examples, and then has a debugging step. And the results are incredible. They get 71% versus a human baseline of 85%, and then he gets 51% prior versus the prior state of the art. And you can see here he says scaling the number of sampled Python rules reliably increased performance up to 3% accuracy for every doubling. And we are still quite far from the millions of samples that AlphaCode uses. Basically what he's stating here is, is that they did really, really well and they didn't Even need to sample millions of samples like AlphaCode did, which we just talked about. And Francis Chollet actually responded, stating that this has been the most promising branch of approaches so far, leveraging an LLM to help with discrete program search by using the LLM as a way to sample programs or branching decisions. This is exactly what neurosymbolic AI is for the record. So this is pretty crazy because even he, the guy who created the benchmark, is stating that this is the right path to go down. And even some of you know, the craziest critics like Gary Marcus, who state that there will be no AGI without neuro symbolic AI. And he's got this entire talk like I was literally listening to this and Basically it's a 30 minute talk in which he discusses, you know, it's, you know, everyone's got the current approach wrong and it's a really, really fascinating, you know, you know, piece of where he talks about, you know, this entire paradigm that we are on is probably, probably wrong. But I'm guessing that maybe now with this new sort of approach that we're doing, things actually might start to get towards human level in nearly every aspect. Now what's actually crazy about this entire thing here is that you can see that in his qualitative analysis there are actually some key things where GPT4O is actually pretty limited. You can see GPT4O is limited by failures other than the reasoning, which we know is, you know, I guess you could say pretty limited in the fact that GPT4O's vision is terrible on grids. When asked to describe what is in somewhat a large grid, it often fails to see the input correctly and states what's wrong, facts about which colors in some location or what's present. In particular, it totally fails to extract the colors of the cells from an image. For images 12 by 12 and is quite bad at 8 by 8. Visual abilities as poor as GPT 4.0, it would often take them quite a bit of effort to even solve simple ARC AGI problems. And if you want a frustrating time, try solving some arc AGI problems without using vision other than reading I.e. try to do them without ever drawing out the grids in 2D, forcing yourself instead just to interact with a textual representation of the data. For hard mode, you could try doing this blindfolded with a friend, allowing you to dictate Python lines of code to run on the image. And I think that this would be quite hard. So basically here he's trying to state that, look, okay, this system that I built that is able to get a state of the art on this ARC AGI evaluation benchmark is very limited by the fact that GPT4O's vision just is innately not that good for this specific task, whereas human vision is really really good. So this is going to be one of the things that probably on future models where the vision systems do get a lot better, we're going to be seeing if the interpretation is gets a lot better as well. Now of course he says GPT4O isn't that good at coding and makes simple mistake like one off, you know, off by one errors extremely often and we don't do multi round debugging because it's probably cheaper and more effective just to get more samples in the current regime. Of course GPT4O sometimes hallucinates, which means that this of course could reduce the reliability of the results. And of course he says GPT4O is worse at using long context than other models. He says I think that the long context for GPT4O is quite bad and starts taking a big hit about after 32 to 40,000 tokens based on my qualitative impression, which is limited by my ability to use longer prompts with more examples and detailed representations. And he says here is that it doesn't seem to respect my few shot prompt and often does somewhat worse stuff than what it should do based on the few shot examples. For instance, it systematically responds with much shorter completions than it is supposed to, even if I give it very specific instructions to do otherwise. So of course you can see here GPT4O the context length. This is something that hasn't really, really increased that much. And this is because OpenAI have, I wouldn't say stalled, but they've not been under the pressure like any of the other AI labs to roll out systems that can intake huge context lengths and of course output huge context length. So they also state here as well that not having flexible prefix caching substantially limits approaches, which is of course something that is going to limit the system. And then of course he talks about removing these non reasoning weaknesses would improve the performance of my solution by a significant amount. Vision is especially a large weakness. So the point here guys, and this should be like not so much of a revelation, but it should be an eye opener for you that this new benchmark is wasn't solved immediately, but the guy was able to get 50% using the current state of the art technology. Okay? And that's even with the fact that there are these very, very obvious limitations to the systems, which means for us that there is still a very, very large room for AI to grow. Because these things aren't like where we've hit a wall. We were like, okay, we have no idea how we're going to improve the vision, we have no idea how we're going to improve the coding, we have no idea how we're going to improve the long context. These are kind of things that I wouldn't say have defined solutions, but people are actively working on them and we can say with a decent amount of prediction that this is going to get better. And once these get better, once you combine them into certain frameworks and certain architectures, like using GPT4O in this context, I think that this is going to probably surpass the ARC AGI benchmark probably by the time GPT5 is released, which would be pretty, pretty fascinating. Doesn't mean it's going to be AGI, but maybe it's going to be better than the average human. So I'll leave a link to this in the description. But of course there are some predictions in here and it's pretty interesting because he says this is a 70% probability that a team of three top research machine learning engineers with fine tuning and access to GPT4O, $10 million in compute and one year of time could use GPT4O to surpass typical naive Mturk performance at ARC AGI on the test set while using less than $100 per problem at runtime. And he says here that There is a 60% probability that if a next generation frontier model like GPT5 was much better at basic visual understanding, for example above 85% accuracy on the vibe eval hard using this same exact method with minor adaptation tweaks as needed, that LLM would surpass typical naive MTurk performance. And basically what he's stating here is that with future systems like GPT5 and there's an 80% probability that the next generation of multimodal models will be able to substantially improve the performance on ARC AGI, which is pretty, pretty incredible. Which means that when the next systems do release, it is very likely that we're going to see a whole lot of new benchmarks broken. Especially with the fact that we've got all of these sort of frameworks that are going to be surrounding this technology. So this is just a quick reminder for those of you who like to support the channel. I recently launched a school community, this is a private community where we actually focus on things like the post AGI framework that I developed. You get instant download access to this where you are just going to have framework that easily allows you to navigate the post AGI economy with ease. Of course my personal strategy for making money with AI, exclusive tutorials on how to actually use no code AI agent frameworks and of course the AGI proof investment deck that's helped me make very very sizable returns. If that interests you, don't forget to check it out. If not, just enjoy the rest of the videos on the channel.