OpenAIs Surprising New Plan For Superintelligence...

So there has been a recent Pastebin floating around, and I don't usually cover Pastebins like this because they aren't usually true, and it doesn't usually help the AI space in terms of development when, you know, there's a lot of leaks floating around. But I wanted to cover this because it does make some claims which haven't really been picked up by the mainstream AI consciousness in terms of how impactful these changes are truly going to be. So I'm going to walk through some of these claims because when we do actually look back at some of the things that AI is doing and look forward some of the things that AI could do. I do think that this Pastebin, which is essentially just a website where you can anonymously paste a piece of text, and we've previously gotten some verifiable leaks from that. But I do think this one is more interesting than it is real, I guess you could say. But we're going to be taking a look because it basically covers how AI achieves superhuman intelligence through. Through video games. So this is a document that is claiming that superhuman intelligence is going to be coming through video games. And this might actually be true because the only times we've truly seen superhuman intelligence is currently through a video game setting. So one of the things that this document claims is that it claims that the superhuman intelligence AI that OpenAI is developing is essentially going to use reinforcement learning. So reinforcement is basically where it receives feedback from the game environment and continuously improves its performance. This iterative process allows the AI to refine its strategies to superhuman levels. Now, basically all that happens is that because AI systems are virtual and they can be trained in virtual environments, what you can do is you can essentially get continuous feedback. So what that means is that unlike humans, where we might make a mistake and it might take us a day or two or even a month to figure out, figure out if that mistake is right or not. An AI can make a million different mistakes a million different times, and it can play itself millions and millions of times as it continues to play its times. It gets rewarded for the good results and gets, you know, punished for the bad results. It can then, you know, know exactly what to do. And in the simulation, it can figure out crazy things. Now, the thing about reinforcement learning is that it has been used before, and with reinforcement learning, it was actually used previously when with OpenAI. So some of you might know this, but OpenAI actually did do some things regarding video games a few years ago. And when I say a few years ago, I mean around five to six years ago. So this was pre ChatGPT era, where they were focusing just purely based on some research stuff. So basically what you can see here, this is OpenAI and Dota 2. So essentially what this was, this was a game where they used reinforcement learning to kind of, I guess you could say, train their models to get better at a video game. And what was crazy was that this video game agent thing managed to destroy and beat the top level world champions, which was pretty crazy. So take a look at the first initial findings when they were doing this, because it does go to show how important reinforcement learning is.

Good game humans no longer think they can win. They would be absolutely correct. So their dev team got absolutely crushed. I think it was the fastest casting gig of my life. Then it went into game number two. The. The humans kind of, I mean, they had time to, like, think about the game and stuff like that. They got crushed even harder. And the bots did exactly. What I hope for is they own this area of the map. You take away two thirds of the map. They didn't even touch these two bottom towers, and they would be 100% correct in this. This is like one of the highest level plays that you can make. This side of the map is incredibly hard for the boss to control. And so they're just playing this top side and this mid side because they understand that these are the two most important parts to control in the game. The ability to, like, intuitively do this is insane. Doing it one game, I could maybe chalk it up to just dumb luck. Doing it two games in a row. Flipping the sides means that it's more than just coincidence. It took me, and I'm fairly reasonably good at the game, eight years before I learned some of the strategies. I would say it was pretty easy to quantify for me. It was about eight years for me to learn the strategies that the bot was intuitively doing.

And I just want to interject here because what he just said shows you how important this is. He said that, you know, it took him eight years to learn what the bots did. And that's the point. Okay, but with reinforcement learning, you can compress years of knowledge into, you know, sometimes just a few hours. And of course, you know, with large clusters, large training systems, it might take a couple days or a couple weeks. But the point is, is that you can compress the time it takes to learn knowledge in a simulated environment. And because it's an AI system, it can do that remarkably well. And he was saying, look, it took me eight years to figure out what's going on. And these bots have already figured out exactly what's going on intuitively.

Doing to train our bots, we use.

Reinforcement learning with self play. We run the game on over 100,000 CPUs and our bots learn from every game they play. Because Dota is so complex to learn, even for a single player, we created a hyper parameter which we call team spirit. The five bots start out completely selfish, but tuning this knob tells them to care about their teammates so that they can learn to play together as one unit. Now, Dota 2 wasn't the only game where OpenAI used reinforcement learning. There was actually also something that broke the Internet at the time because it now has over 10 million views. But this is multi agent hide and seek. And this was another way where they used reinforcement learning. Basically with this one, what they wanted to do was they wanted to take, you know, two teams of AI bots. They wanted to have an AIs that were, you know, the seekers, which are these little guys in red, and then they wanted to have these hiders that were these little guys in blue. They basically did millions and millions of games in this simulated environment right here. And, and they made some pretty interesting discoveries. Take a look.

On Earth, the simple rules of natural selection and competition led to the evolution of increasingly intelligent life forms. Today, we ask if comparably simple rules and multi agent competition can also lead to intelligent behavior. In a new virtual world, these agents are playing hide and seek. These agents have just begun learning, but they've already learned to chase and run away. This is a hard world for a hider who has only learned to flee. However, after training in millions of rounds of hide and seek, the hiders find a solution. The hiders learn to use rudimentary tools to their advantage. By grabbing and locking these blocks, they can create their own shelter. The seekers are locked in place for a brief period at the start of the game, giving hiders a chance to prepare. Even so, the hiders must learn to collaborate, accomplishing tasks that would be impossible for any single individual. The hiders are not the only ones who can learn to use tools. After many generations of failing to break into the shelter, the seekers learn to jump over obstacles using ramps. However, after many millions of rounds of having their shelter breached, the hiders learn to take away the primary tool the seekers have at their disposal. Note that we did not explicitly incentivize any of these behaviors. As each team learns a new skill, it implicitly changes the challenges the other team faces, creating a New pressure to adapt. We've also put these agents into a more open ended environment, randomizing the objects, team sizes and walls. In this world, they learn to construct their own shelter from scratch, requiring that they arrange multiple objects into precise structures. To prevent seekers from using the ramps, the hiders move them to the edge of the play area and lock them in place.

And now here's where things do get really interesting. Because after training it millions and millions of times, these AI agents managed to figure out really unique and innovative ways to solve complex problems. And the crazy thing about this is because you're not explicitly training them to do X or to do Y, just do whatever you want to do and then, you know, you have whatever your reward function or whatever the point is, is that you manage to then get innovative solutions because you're not training for a specific outcome, you're basically just training for it not to be, you know, got by the seeker. And you can do that in a variety of different ways.

We originally believed this would be the final strategy that the agents learn. However, we found that after more training, the seekers discovered that they can jump on top of boxes and surf them to the hider's shelter. In the last stage of emergence strategy that we observed, the hiders learn to lock as many boxes as they can before constructing their fort in order to defend against box surf. So how do agents acquire these skills? These they're trained using reinforcement learning, an algorithm inspired by the way animals on Earth learn. The agents play thousands of rounds of hide and seek in parallel for many days. They train against each other as well as past versions of themselves using an algorithm called self play. Co evolution and competition on Earth led to the only generally intelligent species known to date, humans. While this world is far less complex than Earth, we have found evidence that simple rules can lead to increasingly intelligent behavior from multi agent interaction. We hope that with a much larger and more diverse environment, truly complex and intelligent agents will.

So basically here you can see that they're just talking about how the fact that they've managed to get these systems to evolve and basically simulate exactly what's going on on Earth over millions and billions of years. And I think this is why people are looking at this, because the fact that you can get these emergent abilities that spawn out of nowhere and the fact that you can get these innovative abilities, you know, in things like AlphaGo, which is move 37, where basically because these systems have played millions and millions of times, they can discover, I guess you could say, new knowledge that we haven't looked at before. And you know, one of the things that was going on in this that, you know, wasn't in this video, but some really cool things that they were able to do was that these, you know, AI systems, they were able to literally break the physics of this. So these robots were able to, you know, get these ramps and they were able to throw them out, they were able to glitch them out. They were also able to fly up into the air and jump straight back down. And it was pretty crazy. After a lot more generations and with that physics breaking and all this kind of stuff, some people are arguing that that kind of super intelligence, where you've got all of these new capabilities coming out of a pretty sterile environment, you kind of wonder if that can be applied to real life. And I think that's why that was mentioned in the post. Now, Dr. Jim Fan here is talking about DeepMind's SEMA, which is an agent that plays seven games and four 3D simulations by reading pixels and generating keyboard mouse control. And this is the original promise of OpenAI universe in 2016, which was way ahead of its time. And after eight years this is done with the modern horizon stack. Now, Basically, this is DeepMind SEMA. Now this is basically an AI agent game where it can play the game only for 10 seconds, so it can't play for hours on end. The data pipeline isn't very scalable. You need to have ways for the agent to explore autonomous and look for novel activities to engage. It can't do that much, but it's able to play it by keyboard and mouse, which is a breakthrough in a sense. So the reason that I've added this is because in the document it talks about generalizing skills. The skills and strategies learned in video games are generalized to other domains. For example, the strategic thinking and planning required in games can be applied to fields such as mathematics, science and complex real world problem solving. Now, whilst that might be a stretch to go from a video game to straight solving math equations, I think the concept of generalizing these skills is there. And it's kind of true because when you look at Google's DeepMind SEMA, it says as we expose SEMA to training more training worlds, the more generalizable and versatile we expect it to become. So basically, as they manage to get an agent to work in many different video game environments, it wouldn't be crazy to say that maybe we could use this in robotics in our earth environment. Like, you know, what's to say that? And it says, and with more advanced models, we hope to improve Siemens understanding and ability to act on a higher level language instructions to achieve more complex goals. So you can see here that it's not completely crazy to say that because with more advanced models they're going to basically, you know, use that to have more understanding and you know, act on higher level language instructions and basically do a lot of different sub actions and all those kinds of things now. So you can see here it says but learning to follow instructions in a variety of game settings could unlock more helpful AI agents for any environment. And this is something that is being worked on. AI agents are something that we currently do struggle with due to the generative nature of our current AI systems. It just doesn't work well across many different things. But the point here is that if we manage to get these AI agents to work in video games, then we're probably going to be able to get them for any environment. So it says our research shows how we can translate the capabilities of advanced AI models into useful real world actions through language interface. We hope that seems and other agent research can use video games as sandboxes to better understand how AI systems may become more helpful. So you can see here that this is something that Google is clearly working on. Now what we also do have is generalizing skills. And what was crazy about this is that DeepMind SEMA generalizes better than the models that were explicitly trained just for that task. So for example, it says in our evaluations, SEMA's agents trained on a set of nine 3D games from our portfolio significantly outperformed all specialized agents trained on each individual one, which is pretty crazy because they've been trained specifically for that game and SEMA outperforms them, which is, you know, pretty, pretty crazy. So it says, what's more, an agent trained in all but one game performed nearly as well on that unseen game, which is also pretty crazy. So it wasn't even trained on that game, but it was able to perform better than the one that was trained on that. So it's pretty crazy because basically what they're saying here is that SEMA has the ability to function in brand new environments and it's able to generalize beyond its training, which is a key factor in order for robotics to succeed. One of the reasons that, you know, autonomous self driving and robotics really does struggle is because it's not able to generalize beyond its training data. So if you've got an environment mapped out, which is good, that's fine. But the reason you need a lot more training data is because when you know You're a robot and you see a million different scenes, you're just not able to understand what's going on, and you just struggle with the task. So if you could get a robot that could literally generalize beyond its training environment, it's able to work in brand new environments, which is something that, you know, modern robots struggle with. So, for example, if I asked you to place this cup on a desk, you could easily do that in any environment. You could be in a desert, on a stranded island, and I could say, place this deck on a cup and, you know, place his cup on a desk, and you could absolutely do that. But a robot, you know, you switch around the environment, you switch around the sizes, it's going to start to struggle, you know, when identifying certain things. And if we can get an AI agent that's able to do that, that's going to be an important, integral piece of research. Now, one of the other crazy things as well that I just quickly want to touch on is that if you go back to here, it basically talks about, you know, superhuman intelligence through video games. And, you know, the more and more research I start to do, the more I don't think that that's completely impossible if you transfer those met. Now, recently we did get a piece of information which kind of confirms that superintelligence isn't as far off as we think. And whilst that might be hyped to drum up investment, I don't think it is. Now, this is from Ilya Satskova's company, if you haven't been paying attention, he started something called Super Intelligence, Safe Superintelligence Inc. And I think the biggest thing that people missed about this is the fact that their statement is that superintelligence is within reach, which is a pretty wild statement to make considering the fact that allegedly, externally, we don't have AGI yet. So, I don't know if, you know, OpenAI have achieved AGI internally, but the fact that one of the lead researchers is stating that superintelligence is within reach is, of course, a indicator that maybe we are a lot further along than most people will have thought. Now, it's important because he was working on super alignment at OpenAI, and clearly they've been working on some stuff which leads him to believe that superintelligence isn't that far off. Now, what we have here is, you know, this graph from Leopold Aschenbrenner's document called Situational Awareness. And essentially right here, this is where he talks about how, you know, automated AI research is possible. In around just three to four years. And three to four years is not that much time. Like, if you think about how like your last three years of your life, maybe some things have changed, but I think the economy or the world going through a huge transformative period in just three to five years, I think that's going to be monumental. And whilst, yes, it may take some time to distribute the technologies, and even if breakthroughs are made, you have to do testing, manufacturing, all those kinds of things, compute building, energy storage, yada, yada, yada, all of those things that you need to do to ensure you can actually get to super intelligence. The point is, is that it's something that is no longer this sci fi trope. It's something that is possibly a reality considering where we've come in the past. And once again, Leopold Aschenbrenner is someone that was working on super alignment at OpenAI. So this is someone that was working on aligning super intelligent. So this is something that just isn't from a random blog post from a random blogger. This is from someone that is arguably got the most knowledge on where we are with regards to superintelligence. So that trajectory right there isn't like clickbait or anything like that. But I do think that this will probably be one of the most important graphs to watch in the future, because if we look at where we're going to be in 2030, it's going to be pretty insane on the kinds of systems that could be built. And that's not just by OpenAI. This is by any developing nation with a race to develop their own frontier models. Now, something else that this document dives into that I do want to cover is the Monte Carlo Tree Search Evaluation. Now, Monte Carlo Tree Search is a very popular tree search method which evaluates the strategies, then picks the best one based on simulations. Then it uses a search tree to represent the possible game states and actions, and then it runs a simulation to determine the best strategy. So basically, if you're a human doing this, you would basically think about a scenario, think about all the possible scenarios, try and run a simulation that could run the scenarios. Once you see the outputs of those scenarios, you think, okay, I'm not going to go ahead and do that. I'm going to go ahead and pick the best one. So this is basically how it works in simplistic terms. And yeah, it's something that does work a lot because they use this in AlphaGo. So you can see here that the trained network is used to guide a search algorithm known as Monte Carlo tree search, mcts to select the most promising moves in games. For each move, Alpha 0 searches only a small fraction of the positions considered by traditional chess engines in chess, for example, it searches only 60,000 positions per second in chess, compared to roughly 60 million for stockfish. And it managed to beat Stockfish. So the crazy thing here is that we have a system that is searching only ten thousands of moves, but is able to get a lot better than them. Which means that the search architecture is of course, something that advanced AI systems will need to do. And, and this is something that humans do automatically. If you're someone who's about to make a decision, especially in chess, you tend to think about all the different kinds of outcomes that may result. You tend to think about 100 or 50 different outcomes, and then you tend to think, okay, provided all of those outcomes, and provided the outcomes of the outcomes, what position am I going to be in? You then tend to get a very good grip. And the thing is, why they say human brain is so great is because human, humans manage to have a search program that works on only hundreds of moves, but for current AI systems, they have to search through so many moves in order to get that right decision, which is pretty crazy. Now, this isn't just theory, of course. Like I said, this is something that happened in alphago, but of course, this is spoken about clearly. And I've referenced this clip before.

These foundation models are world models of a kind, and to do really creative problem solving, you need to start searching. So if I think about something like AlphaGo and the Move 37, the famous Move 37, where did that come from? Did that come from all its data that it's seen of human games or something like that? No, it didn't. It came from it identifying a move as being quite unlikely, but possible, and then via a process of search, coming to understand that that was actually a very, very good move. So you need to get real creativity. You need to search through spaces of possibilities and find these sort of hidden gems. That's what creativity. I think current language models, they don't really do that kind of a thing. They really are mimicking the data. They're mimicking all the human ingenuity and everything which they have seen from all this data that's coming from the Internet that's originally derived from humans. If you want a system that can go be truly beyond that and not just generalize in novel ways. So it can, you know, these models can blend things. They can do, you know, Harry Potter in the style of a Kanye west rap or something, even though it's never happened, they can blend things together. But to do something that's truly creative, that is not just a blending of existing things, that requires searching through a space of possibilities and finding these hidden gems that are sort of hidden away in there somewhere. And that requires search. So I don't think we'll see systems that truly step beyond their training data until we have powerful search in the process.

So that was Shane Legg, who is someone who states that AGI by 2027 and he is the co founder and chief AGI scientist at Google DeepMind. So his opinion is very important when it comes to discussing these kinds of topics. But the point here is that what we have is a situation that shows us that this is truly going to be an effective method. And Samaltman has actually spoken about this how, you know, GPT4, if you get that model to search over many different things, you end up basically with a lot more responses, well, better responses because you can find out which ones are gonna be there. Now something that was also kind of interesting and that relates back to this search was, you know, neurosymbolic AI. So essentially neurosymbolic AI. This process combines neural networks with symbolic reasoning, enabling AI to handle abstract concepts and logic effectively. Gaming experiences helped develop these advanced cognitive abilities. So essentially neurosymbolic AI just means that you're giving an AI a piece of a system that it never really had before. That kind of completes the system in the way that humans are complete in their part. So basically the neural networks part, the neuro part, is basically the robot's brain that learns from examples. Think of it like a child learning what a cat looks like by seeing lots of pictures of cats. And this part is great because it can recognize patterns and, you know, understands things that it has seen many times. Now the symbolic part, this is where the robot's library of rules comes into play. This is like teaching the robot specific instructions, like how to play a game by following the rules step by step. And this part is good at thinking logically and following clear instructions. So when you combine these things, okay, this is where we get AI systems that are really effective. And many people have said that, you know, your never going to get to AGI unless you have neurosymbolic AI. And basically combining the power of neural networks with the logical thinking of symbolic AI, this is how you get to truly advanced systems. And this is what we had with AlphaGo. AlphaGo was primarily based on neural network technology, but it also incorporated elements that can be considered part of a symbolic AI. And what was interesting about this entire thing about AlphaGo Neurosymbolic AI was that this isn't the only thing that had this because it's a symbolic component of Monte Carlo's research, searching through millions of different possibilities that was actually recently done with this. So basically, Francis Chollet is someone who recently spoke about this ARC AGI test. It's basically a test in which if an LLM can't do it, it's not AGI yet and it's not smart yet. It's basically tries to test how well an AI can actually reason, because the tests that it's doing aren't really things that are in the training data at all. So essentially with Francis Chollei, he's responding to a tweet here where someone actually used an LLM with discrete search. So basically, in order to complete his test, this ARC AGI test, someone used a search method and they managed to get 50%. Now you might think 50%, that's not even that good. But the thing is, before that, people were saying that this was never going to be done and 80% was what they're claiming to be an actual AGI system. So he says that this has been the most promising branch of approaches so far, leveraging an LLM to help with discrete program search by using the LLM as a way to sample programs or branching decisions. This is exactly what neurosymbolic AI is, for the record. And here he states, of course, AlphaGo was neurosymbolic as well. And basically the point here is that asking these AI systems, these LLMs to do a lot of things, it doesn't really make sense. But getting these AI systems to be combined with things like code interpreter or whatever, tree search, this is where we're truly going to get systems that are truly incredible. Now, Sam Altman did briefly kind of speak about this a little bit, but he kind of hinted towards the fact that you need AI systems that think in a different way. So it's kind of interesting. I just want to put this here. And of course this covers the Q thing. There's just some questions I would love to ask your intuition about what's GPT.

Able to do and not so it's allocating approximately the same amount of compute.

For each token it generates.

Is there room there in this kind of approach to slower thinking, sequential thinking? I think there will be a new paradigm for that kind of thinking.

Will it be similar architecturally as what we're seeing now with LLMs, is it a layer on top of the LLMs?

I can imagine many ways to implement that. I think that's less important than the question you were getting at, which is, do we need a way to do a slower kind of thinking where the answer doesn't have to get. I guess spiritually, you could say that you want an AI to be able to think harder about a harder problem and answer more quickly about an easier problem. And I think that will be important.

Is that like a human.

All kinds of research. We have said for a while that we think better reasoning in these systems is an important direction that we'd like to pursue. We haven't cracked the code yet. We're very interested in it.

And what's interesting about all of this is that one of the notable AI skeptics, Gary Marcus, even agrees with this term. So the reason why I'm talking about Gary Marcus is because basically everything that has been popular this year has been a lot of AI stuff. And he's been a huge critic of the current AI paradigm, which is LLMs and transformers. And he basically says that, you know, you need neurosymbolic AI. Now, I actually like Gary Marcus because I always state that, you know, if everyone's thinking in the same direction, then absolutely no one is thinking, and it's always good. Even though, you know, sometimes he can be a bit too critical of AI. I do think that opinions like this are important because if you start to think, wait a minute, this guy might actually be right. And maybe he's not just a critic, but maybe he's hinting at a clear limitation of the models. And if we actually listen to what he's stating and actually work on that area, maybe we'll actually get AGI. I think it's important to what he's stating here. Okay, he's stating that, you know, neurosymbolic AI is most certainly needed.

Wait, I didn't get that. Okay, so I have three last slides, if I can afford it then. So summary 1. Scaling alone is not all you need. I wrote this slide a couple years ago and it's still true. I didn't change the word current. Deep learning still struggles with reasoning and factuality, even giving vast amounts of data enormous models. I should should say even vaster amounts of data hasn't changed. Classical AI isn't the answer either. Like, I'm not expecting that pure symbolic AI is going to solve all this.

We.

We don't know how to generate knowledge at fast enough clip and so forth. There are lots of problems we have to combine online learning with abstraction. Neurosymbolic approaches, I think offer our best hope here. There shouldn't be 50 people in this room, there should be 5,000. Like if, if we had a tenth of the resources for Neurosymbolic AI as we do right now for large language models, it complet the world. But advances in neurosymbolic AI alone won't be enough because intelligence is multifaceted and we shouldn't expect any one science fits all solution here. Which is why, going back to your, you know, three year problem that I was teasing you about, like we're just not going to solve all this in.

The next few years.

There are many facets of things. We also need large scale knowledge, including some in need.

So yeah, with that being said, take a look at this diagram. And although this diagram is inaccurate in terms to language and reading, because there are many different parts of the brain to where, you know, language actually corresponds to. But the point here is that when you, if you're actually trying to represent the human brain, maybe the human brain isn't entirely intelligent. But the point is that LLMs are only one subsection of human intelligence and there's actually a lot more that needs to go into the pipeline in terms of neurosymbolic AI. With that being said, hopefully you all enjoyed this video and I'll see you on the next one.