New Research Proves AGI Was Achieved...

Did the AI community just pass the AGI threshold and not even realize it? That's what this paper is aiming to look at, which is called the Surprising effectiveness of Test Time Training for Abstract Reasoning. Now, this is research coming out of mit, and I think this is going to be a really fascinating paper because it discusses one of the hardest benchmarks that exists in AI. Most of you guys know about GSM8K GPQA, those benchmarks, but did you know that there was a specific benchmark invented by Francis Soleil, who is a senior staff engineer at Google, who's best known for creating the Keras Deep learning library in 2015. Now take a look at what he says about the ARC AGI benchmark, because when I show you the results of his recent research, you're going to understand why this research is important. What is the ARK benchmark? And why do you even need this prize? Why won't the biggest LLM we have in a year be able to just saturate it?

Sure.

So.

So ARC is intended as a kind of IQ test for machine intelligence. And what makes it different from most LLM benchmarks out there is that it's designed to be resistant to memorization. So if you look at the way LLMs work, they are basically this big interpolative memory. And the way you scale up their capabilities is by trying to cram as much knowledge and patterns as possible into them. And by contrast, ARC does not require a lot of knowledge at all. It's designed to only require what's known as core knowledge, which is basic knowledge about things like elementary physics, objectness, counting, that sort of thing, the sort of knowledge that any four year old or five year old possesses. Right. But what's interesting is that each puzzle in ARC is novel, is something that you've probably not encountered before, even if you've memorized the entire Internet. And that's what makes it. Sorry, that's what makes arc challenging for LLMs. And so far LLMs have not been doing very well on it. In fact, the approaches that are working well are more towards discrete program search.

So essentially, if you didn't understand what Francis Chollet just said there, he's basically saying that, look, this ARC benchmark that he invented is essentially one that is a lot different to traditional benchmarks that allow for LLMs to excel. And even if they've already seen the question, and this kind of testing is vastly different because it means that you can't train for this kind of exam. It has to be within you to reason and understand. And humans perform at around 85%. But LLMs really do suffer. So for those of you who want to see what this test kind of looks like, this is exactly what the test looks like. Now it's actually pretty simple in this regard. You can see that there are holes in this one. You can see that the yellow area is filled in this area. Then on this one, all the areas where you have an object with a hole in the middle, it's filled in with yellow. Then for here you can see the same, it's yellow. And then for here you can see that of course that would be yellow. That would be yellow. All of those areas you'd fill in for yellow. And of course you do that for the output. Now the thing is that LLMs do struggle with these kind of tests because they haven't actually seen them before. Which means that when it comes to reasoning over problems that they haven't seen before, LLMs do struggle. And this is what you call the problem of LLMs struggling with things that are out of distribution. And of course, if we are to get to AGI, we need a system that can perform well on these kinds of tests because it means that they will perform well on things that they haven't seen before, which allows them to be a lot more reliable in many different use cases in many different industries. And so that's where this research comes in from MIC that says the surprising effectiveness of test time training for abstract reasoning. It talks about, you know, how language models have shown impressive performance on tasks within their training distribution, but often struggle with novel problems requiring complex reasoning. And we investigate the effectiveness of test time training, which is updating the model parameters temporarily during inference using a loss derived from input data, yada yada, yada, basically saying that, look, we've managed to find a way to increase these models significantly and the results of this are pretty outstanding because they surpass human level reasoning, which is pretty incredible because this is the first time it's been done on a benchmark that traditionally LLMs are considered to fail at. So this was the kind of thing that they did. This is where you can see a quick example of the training test time data. I'm not going to bore you guys with this too much. Essentially this is where they just used a method which is kind of like a method of searching through the possible solutions to a question. And essentially the kind of methods that they did was they flipped the model vertically, horizontally. And one thing that they also did was they left one out. So, for example, let's say you were trying to Predict which number comes in a sequence of 2, 4, 6. Of course 2 4, 6 would be 8. What they did was they then looked at the predictions for 4 and 6 to see which would come before that, which, which is of course two. Then they looked at two to six, what would come between that, which would of course be four. Then they looked at basically different combinations to predict which would come next. And this was basically the variation of their search algorithm which allowed them to search in a way over the possibilities of possible solutions. And then after they essentially generated multiple predictions from these transformed version, the hierarchical voting method aggregates these predictions. The model uses intra transformation voting first, followed by global voting to choose the most consistent and correct answers, likely correct answers. And of course they actually mentioned that they used self consistency to validate predictions across the transformed inputs, ensuring the chosen answer is the one that appears the most frequently across the variations. And this resembles a search for agreement or consistency across the outputs. Now you might think, okay, it's all well and good having all of this crazy stuff and searching over the possibilities, but what were the results of this study? Well, the crazy thing about this and why some people are now claiming that, look, we, we have just slowly approached Eiji and we're basically now the frog in boiling water, which is an analogy to where if you put a frog in extremely hot water, it's going to jump straight out. But if you put a frog in water that is slowly getting hotter and hotter, it won't realize and will eventually boil. And that's apparently what has happened today, where they're basically stating that, look, we get state of the art public validation accuracy of 61.9% matching the average human score. And our findings suggest that explicit symbolic search is not the only path to improved abstract reasoning in these models. Now what's crazy about this guys, is that this is of course state of the art and it's one of the first times where it's reached the human level score, which is completely unheard of when we're looking at a system that is performing on the benchmark that is supposed to be the kind of benchmark that tells us whether or not we have AGI. Now of course some people would argue that this is a kind of abstract reasoning test that doesn't really deduce if we have AGI or not. Because when we look at OpenAI's definition of AGI, AGI is a system that is apparently an autonomous system that outperforms humans at most economically valuable work. Of course there are various definitions, but I do think that even if we do get the same kind of methods applied to different models, we can understand how to make these systems a lot more accurate and, and then translate that into valuable work. Now I do think one of the things here is really interesting because what it's showing us is that there is quite likely a clear path to AGI. And now a lot of the things that we've seen are going to make sense. So let me explain to you exactly what I'm talking about. So one of the things that everyone is familiar with by now is quite likely the O1 paradigm. Okay? And this paper was quite like the O1 paradigm because OpenAI's O1 model searches during inference time. Now the crazy thing is that we don't actually know what OpenAI's O1 models are doing at inference time because these models reasoning tokens are actually hidden from us in order for them to protect their moat. But what we do know is that as test time compute increases, so as you allow the model to think for longer, the ability to get higher scores on benchmarks and reason more effectively increases. And that's exactly what we saw with this paper, which shows a 6 times improvement only using an 8B parameter LLM. Now what's crazy about this O1 paradigm is what it actually reveals to us about some of the data that we've known about AI before. Do you remember AlphaGo and what the creators of AlphaGo were saying about the future of LLMs literally just one year ago?

I think that's on the right track. I think there is. These foundation models are world models of a kind and to do really creative problem solving, you need to start searching. So if I think about something like AlphaGo in the move 37, the famous move 37, where did that come from? Did that come from all its data that it's seen of human games or something like that? No, it didn't. It came from it identifying a move as being quite unlikely but possible and then via a process of search coming to understand that the that was actually a very, very good move. So you need to get real creativity. You need to search through spaces of possibilities and find these sort of hidden gems. That's what creativity is. I think current language models, they don't really do that kind of a thing. They really are mimicking the data. They are mimicking all the human ingenuity and everything which they have seen from all this data that's coming from the Internet, that's originally derived from humans. If you want a system that can go be truly beyond that and not just generalize in novel ways. So it can, you know, these models can blend things. They can do, you know, Harry Potter in the style of a Kanye west rap or something, even though it's never happened, they can blend things together. But to do something that's truly creative, that there's not just a blending of existing things that requires searching through a space of possibilities and finding these hidden gems that are sort of hidden away in there somewhere. And that requires search. So I don't think we'll see systems that truly step beyond their training data until we have powerful search in the process.

And that's incredible because that's exactly what we're seeing with O1. And of course today that's exactly what we're seeing on the surprising effectiveness of test time training for, for abstract reasoning, which is supposed to be the kind of benchmark that LLMs completely fail at. But looking at these methods, it's clear that when you use these kind of search methods and techniques, you're able to bring that benchmark up even higher. Now, Shane leg from Google DeepMind wasn't the only person that actually spoke about this. If you're actually wondering, we actually have information from people who actually worked on the actual O1 model. And look at what he says about a previous game called Hanabi and how the drastic performance increase led to something that they literally couldn't believe. And it was of course due to the search.

This is what you would get by adding this search algorithm to the different bots. So if you take this handcrafted heuristic bot that was only getting 28% and then added the simplest search imaginable where you just like, you know, do a bunch of rollouts for all the different actions you could take and then pick the one that had the highest expected value that would boost your performance to nearly 60%, which was beating all the previous Deep RL bots just out of the box. This was using like a single CPU core at test time for like a second. And the beautiful thing was that you could actually add this on top of all the other Deep RL bots. So if you added it to like the latest and greatest bot, Deep RL bot, you would boost the performance even further to around 72%. And then if you did this, this was only if you did search for a single player. So if you did it for both players, that's the green bars and you can see the performance went up even more. Now I should also point out that the point that the upper bound for this game is not 100% because there are some, there are some like deal outs that you just cannot win. So really the top performance is possible is like I think maybe 90%. And so you can see like we're quickly saturating performance in this domain. Now when my teammates and I at fair got this result, my, my teammate literally thought it was a bug because it was just unimaginable that you do this like simple thing and the performance jumps up from like 28% to state of the art 58%.

So this is why there is a clear trend that we're going to move towards an area where it looks like test time compute and train time compute are going to be things that are used to allow these LLMs allow these AI systems whichever way, whichever mechanism they decide to use in terms of their searching. Because there are a variety of different ways that you can actually search with an AI system. But it's quite clear that this is going to be the next method that a lot of future systems are going to have in order to unlock advanced reasoning and of course in order to unlock the ability to perform things that are out of distribution. Now what's crazy about this is search is incredible. But this actually got me thinking, okay, because when we look at how humans are versus how AI systems are, sure you can have an AI system that searches over a thousand possibilities or even 10,000 possibilities, but what happens if that search were to get even more sample efficient? Take a look at what Demis Hassabis says, because this is exactly what I'm thinking about. Sure we can get superhuman AI systems, but that kind of search just isn't as efficient as humans are.

Would maybe look at millions of possible moves for every decision it's going to make. AlphaZero and AlphaGo made, you know, looked at around ten tens of thousands of possible positions in order to make a decision about what to move next. But a human grandmaster, a human world champion, probably only looks at a few hundreds of moves, even the top ones, in order to make their very good decision about what to play next. So that suggests that obviously the brute force systems don't have any real model other than the heuristics about the game. AlphaZero has quite a decent model, but the, but the human, you know, human top human players have a much richer, much more accurate model than of go or chess. So that allows them to make, you know, world class decisions on a very small amount of search. So I think there's still, there's a sort of trade off there. Like you know, if you improve the models, then I think your search can be more efficient and therefore you can get further with your search.

Yeah. Now when you start to understand that, okay, this now makes sense. These AI systems are basically searching through other variety of different possibilities with a more creative output set via the temperature. And this means that we're allowed to get a lot of different solutions that have a lot more variety, thus allowing us to get to the right answer more often. Of course, you train on those thought patterns that led to those right answers, but you might be thinking, okay, since this is now the case and since this is here, I'm thinking that this is probably why Sam Altman said in a recent interview, this is why they know exactly what they need to do in order to get to AGI. This is the first time ever where.

I felt like we actually know what to do.

Like, and then of course, when we do, remember that in OpenAI's blog post where they spoke about 01, they spoke about how when O1 was allowed 10,000 submissions per problem, the model achieved a score of 362.14, which is above the gold medal threshold. And this is of course the International Olympiad in Informatics. And this is an incredible feat for any LLM. But you can see that now with this kind of added search and submissions, we're able to get to that gold medal threshold. Due to these statements from this research paper and a variety of other different sources, it now leads me to believe that a lot of the information coming out of OpenAI isn't hype anymore. It does seem like they've got the roadmap to AGI, which might just be a combination of different search tactics, as well as refining how they manage to make that search even more efficient. But it will be interesting to see what even happens with the O2 model. As Sam Altman is predicting, it gets 105% on the GPQ. He also predicted that it will saturate a lot of these benchmarks, but a lot of other companies research are basically supporting the claims for the 12 paradigm. I'd love to know your thoughts on whether or not we have achieved AGI in I guess the out of distribution aspect, but it will be interesting to see where things go from here.