Hassabis, Altman and AGI Labs Unite - AI Extinction Risk Statement [ft. Sutskever, Hinton + Voyager]

And Geoffrey Hinton, Here is what these statements said. Mitigating the risk of extinction from AI should be a global priority alongside other societal level risks such as pandemics and nuclear war. It is now almost impossible to deny that this is now the consensus view among AI experts. Let's first look at the preamble, then break down the statement and show you the signatories. They say that AI experts, journalists, policymakers and the public are increasingly discussing a broad spectrum of important and urgent risks from AI. Even so, it can be difficult to voice concerns about some of the advanced AI's most severe risks. The succinct statement below aims to overcome this obstacle and open up discussion. It is also meant to create common knowledge of the growing number of experts and public figures who also take some of advanced AIs most severe risks seriously. The first point is that the statement is in a way optimistic. It says we can mitigate this risk, perhaps not eliminate it, but mitigate it, reduce the risk. Second, it says we should do this globally. And that's not just among all the different AGI labs, almost all of which signed the statement, but also between countries. In that vein, there were quite a few prominent signatories from China. And the third point that I'd make is that they put it on a par with pandemics and nuclear war. Toward the end of the video I'll show you. That's not as far fetched as it sounds. Anyway, who actually signed this statement? Let's find out. We have two of the three founders of deep Learning. That's Geoffrey Hinton and Joshua Bengio. The third founder was Yann Lecun, and we'll touch on him later in the video. All three of those won the most prestigious accolade in computer science, which is the Turing Award. Then we have three of the CEOs of the top AGI labs. Sam Altman, Demis Hassabis and Dario Amadei of OpenAI, Google, DeepMind and Anthropic. None of those signed the pause letter, but they did sign this statement. And actually, as interestingly for me, so did Ilya Sutskova, who I see as the brains behind OpenAI course, also worked with Geoffrey Hinton on deep learning and is widely regarded as the smartest guy in machine learning. You will also notice so many Chinese signatories, especially from Xinhua University, which I've actually visited in China. It's one of their leading universities. That's a really encouraging sign of cooperation between the west and countries like China on AI. And the list of significant signatories goes on and on and on. These are senior people at the top of DeepMind, Anthropic and OpenAI. And there are names like Stuart Russell, who wrote the textbook on AI, who also signed the pause letter. Let me highlight a few more names for you. Here you have the CTO of Microsoft itself, Kevin Scott. He's the guy who basically heads up the partnership between OpenAI and Microsoft. I think many people will miss his name, but I think it's particularly significant that he also signed this notice. Also, the CEO of Stability AI, Imad Mostak, the Center for AI Safety coordinated this effort, and I'll get onto their eight examples of AI risk in a moment. But first let's pick out a few more names. You've got David Chalmers, Daniel Dennett, Lex Friedman, and Victoria Krakovna. Now together with the statement the center for AI Safety also put out this eight examples of AI risk. I've read almost every paper linked to in these eight examples, so I'm going to try to summarize them fairly briefly because I know not everyone will be that interested. It starts by saying that AI could be profoundly beneficial, but also present serious risks due to competitive pressures. Before we get to the risks, I want to touch on some of the upsides recently outlined by Demis Hassabis, and these showcase what can happen if we get this right.

We had sort of a golden couple of years in some sense with AI for science. We've had lucky enough to have many Nature and science papers published in all sorts of domains. So from quantum chemistry, better DFT functions approximate Schrodinger's equation, pure mathematics. We solved some important conjectures in topology with collaborating with some brilliant mathematicians. We found work in fusion reactors with EPFL on their test fusion reactor, controlling the plasma in real time in their fusion reactors and being able to hold the plasma safely in place for arbitrary amounts of time, being able to predict rainfall many hours ahead more accurately than current meteorological models. And then in applications, there's a ton of those too. We saved One of the early things we did at Google was saving about 30% of the cooling energy used to cool the massive data centers. At Google there's a huge energy saving and we're starting to explore doing that across actual whole power grids.

And this echoes what Yoshua Bengio said in a recent blog post, which is that we can build immensely useful AI systems that are modeled after ideal scientists and do not act autonomously in the real world. Yann Lecun recently said that we would never give Current, current LLMs agency, there.

Is a flaw in current autoregressive LLMs. So there is no persistent memory, first of all. But second of all, you cannot control the system, you cannot impose constraints on it like be factual, be understandable by a 13 year old. And that makes them very difficult to control and steer. And so that creates some fears because people are kind of extrapolating. If we let those systems do whatever we connect them to Internet and they can do whatever they want, they're going to do crazy things and stupid things and perhaps dangerous things and we're not going to be able to control them and they're going to escape our control and they're going to become intelligent just because they're bigger, right? And that's nonsense first of all, because this is not the type of system that we are going to give agency to.

That was a week before this paper was published on the results of giving agency to current large language models. The paper showed that current LLMs with agency are able to utilize the Learn Skill library in Minecraft to solve novel tasks from scratch. Zooming into the diagram, you can see how this Voyager model outperforms reflection which I've talked about in previous videos and autogpt. And it discovers new items and skills continually by self driven exploration, significantly outperforming the baselines. Indeed, Andrej Karpathy responded to this study saying very clear that AGI will mega transform society, but still will have. But is it really reasoning? How do you define reasoning? Oh, it's only predicting the next token. Can machines really think? And he called that armchair philosophy. Previously though, even Yann Lecun has admitted some risks, saying, you know, it's like rockets. You test it, it blows up, you tweak it and then try again. I'm not sure I'm okay with an attempt at AGI blowing up the first time, but I'll leave that up to you to decide. So what are these eight examples of AI risk that the center for AI Safety who organized the statement list out? They say that AI systems are rapidly becoming more capable. They can power autonomous weapons, promote misinformation and conduct cyber attacks. As we've seen, they are increasingly able to act autonomously. Now there is so much to say here and I've read each of these, but I want to keep it to just the highlights. So let's move on to the first example, Weaponization. Malicious actors could repurpose AI to be highly destructive, presenting an existential risk in and of itself and increasing the probability of political destabilization. They talk about aerial combat and then building chemical weapons, which I mentioned in in my previous video on governing superintelligence. Then they mentioned developing AI systems for automated cyber attacks. They mentioned military leaders discussing giving AI systems decisive control over nuclear silos. I'm going to quickly try to demonstrate why that kind of autonomous AI might not be such a good idea. I want you to meet a hero. Stanislav Petrov. He was the duty officer at the command center for the OKO Nuclear Early Warning system. When the system reported that a missile had been launched from the US followed by up to five more. Petrov judged the reports from the system to be a false alarm. And his subsequent decision to disobey orders against Soviet military protocol is credited with having prevented an erroneous retaliatory nuclear attack on the US which it says could have resulted in a large scale nuclear war which could have wiped out half the population of these countries involved. An investigation later confirmed that the Soviet satellite warning system, the Mach behind it, had indeed malfunctioned. I would not have wanted that system to be autonomous. Then we hear that GPT4 was able to autonomously conduct experiments and synthesize chemicals in a real world lab. Again, I covered that paper at the time. And then linking back to Petrov. They say an accident with an automated retaliation system could rapidly escalate and give rise to a major war. But that unlike previous weapons, AI systems with dangerous capabilities could be easily proliferated through digital means. Hopefully you can start to see why we need to balance risks with opportunities. But let's move on to misinformation. I think we can all agree that we already have too much misinformation. So let's move on to the next one which is proxy gaming. This has already been showcased in the social dilemma where AI recommender systems are trained to maximize watch time and click rate metrics. And this can lead people into echo chambers that helps develop extreme beliefs in order to make those people easier to predict by the AI recommender systems. So you might think it will be simple just to tell the AI to promote happiness or economic growth. But that might not work out as you intend. Next is enfeeblement. If we delegate more and more tasks to machines, we become increasingly dependent on them. And here they actually mentioned the film Wall E, which if you remember the ending, features this quite comically. Imagine if it becomes well known that companies led by AI CEOs bring in more profit. Well then it wouldn't take long for all companies to be under immen pressure to make their managers and CEOs. AI. And I know what many people will be thinking, couldn't that be an improvement on the current system? And while I know exactly what you mean, in the current world, realistically it would still be the people owning the company that would derive the profit. And while the ultimate answer may be some form of universal basic income, we do need some time to set that up. And the current accelerated AI arms race doesn't give us much of that time. Next is value Lock in, which links very much to the last point about giving small groups of people a tremendous amount of power. In other words, if you want massive change to the way the world works, giving current leaders AGI might not be the best way of doing it. They say that such AI systems might enable regimes to enforce narrow values through pervasive surveillance and oppressive censorship. Next is emergent goals. This is sometimes called misalignment, and we've already seen many AI agents develop goals such as self preservation. And you can see why even a system designed to do good might have that goal. You can't do good and help the world if you're shut down. So it makes sense that even the most benign AI might want to preserve itself and to take actions, including through deception, to make sure that it's not shut off. And this is not just theory. The accompanying academic paper, natural selection favors AIs over humans, gave this example. Agents could behave one way during testing and another way once they are released to win the war Game diplomacy, which many of you will have heard of, players need to negotiate, form alliances, and become skilled at deception to win control of the game's economic and military resources. AI researchers have trained Meta's AI agent Cicero, an expert manipulator, to do the same. In summary, it would cooperate with a human player, then change its plan and backstab them. In future, these abilities could be used against humans in the real world. Again, that's not because they're malevolent or hate humans. It just makes sense. It's smart to do so. This brings us neatly onto deception, and they give the great example of Volkswagen, who program their engines to reduce emissions only when being monitored. And future AI agents could similarly switch strategies when being monitored and take steps to obscure their deception from monitors. Once deceptive AI systems are cleared by their monitors, or once such systems can overpower them, these systems could take a treacherous turn and irreversibly bypass human control. I talked a bit more about that point of when AI might become deceptive in my previous video on governing superintelligence. It is a key debate in the AI alignment community about whether models will become deceptive before they become helpful for alignment. But finally we have power seeking behavior and this example ends on this dark note. Building power seeking AI is also incentivized because political leaders see the strategic advantage in having the most intelligent, most powerful AI systems. For example, Vladimir Putin has said whoever becomes the leader in AI will become the ruler of the world. So those were the eight examples. And yes, I would have signed this statement, but I'm not a significant figure so I can't. Anyway, let me know in the comments if you agree that this should be a global priority. And of course you can also let me know if you don't think it should be a global priority. My goal in this channel is to cover both the risks and opportunities, so I'd love to hear from you whatever your opinion is. Have a wonderful day.