'Pause Giant AI Experiments' - Letter Breakdown w/ Research Papers, Altman, Sutskever and more

Less than 18 hours ago this letter was published calling for an immediate pause in training AI systems more powerful than GPT4. By now you will have seen the headlines about it waving around eye catching names such as Elon Musk. But I want to show you not only what the letter says, but also the research behind it. The letter quotes 18 supporting documents and I have either gone through or entirely read all of them. You'll also hear from those at the top of OpenAI and Google on their thoughts. Whether you agree or disagree with the letter, I hope you learned something. So what did it say? First, they described the situation as AI labs locked in an out of control race to develop and deploy ever more powerful digital minds that no one, not even their creators, can understand, predict or reliably control. They ask, just because we can, should we automate away all the jobs, including the fulfilling ones? And other questions like should we risk loss of control control of our civilization? So what's their main ask? Well, they quote OpenAI's AGI document at some point it may be important to get independent review before starting to train future systems and for the most advanced efforts to agree to limit the rate of growth of compute used for creating new models. And they say, we agree that point is now. And here is their call. Therefore we call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT4 that they are not saying shut down GPT4, just saying don't train anything smarter or more advanced than GPT4. They go on. If such a pause cannot be enacted quickly, governments should step in and institute a moratorium. I will come back to some other details in the letter later on, but first let's glance at some of the eye catching names who have signed this document. We have Stuart Russell, who wrote the textbook on AI, and Joshua Bengio who pioneered deep learning. Among many other famous names, we have the founder of Stability AI which is behind stable diffusion. Of course I could go on and on, but we also have names like Max Tegmark, arguably one of the smartest people on the planet, and if you notice below, plenty of researchers at DeepMind. But before you dismiss this as a bunch of outsiders, this is what Sam Altman once wrote in his blog. Many people seem to believe that superhuman machine intelligence would be very dangerous if it were developed, but think that it's either never going to happen or definitely very far off. This is sloppy, dangerous thinking. And a few days ago on the Lex Friedman podcast he said this I.

Think it's Weird when people, like think it's like a big dunk that I say, like, I'm a little bit afraid. And I think it'd be crazy not to be a little bit afraid. And I empathize with people who are a lot afraid. Current worries that I have are that there are going to be disinformation problems or economic shocks or something else at a level far beyond anything we're prepared for. And that doesn't require super intelligence, that doesn't require a super deep alignment problem in the machine waking up and trying to deceive us. And I don't think that gets enough attention. I mean, it's starting to get more, I guess.

Before you think that's just Sam Altman being Sam Altman, here's Ilya Sutskova, who arguably is the brains behind OpenAI and GPT4. As somebody who deeply understands these models.

What is your intuition of how hard.

Alignment will be like?

I think with the. So here's what I would say. I think with the current level of capabilities, I think we have a pretty good set of ideas of how to align them. But I would not underestimate the difficulty of alignment of models that are actually smarter than us, of models that are capable of misrepresenting their intentions.

By alignment, he means matching up the goal of AI systems with our own. And at this point I do want to say that there are reasons to have hope on AI alignment, and many, many people are working on it. I just don't want anyone to underestimate the scale of the task or to think it's just a bunch of outsiders, not the creators themselves. Here was a recent interview by Time magazine with Demis Hassabis, who many people say I sound like he is the founder, of course, of DeepMind, who are also at the cutting edge of large language models. He says when it comes to very powerful technologies, and obviously AI is going to be one of the most powerful ever, we need to be careful. Not everybody's thinking about those things. It's like experimentalists, many of whom don't realize they're holding dangerous material. And again, Emad most I don't agree with everything in the letter, but the race condition ramping as H1 hundreds, which are the next generation of GPUs come along is not safe for something the creators consider as potentially an existential risk. Time to take a breath, coordinate and carry on. This is only for the largest models, he went on, that these models can get weird as they get more powerful. So it's not just AI outsiders but what about the research? They cite those 18 supporting documents that I referred to. Well, I read each of them. Now for some of them I had already read them, like the Sparks report that I did a video on and the GPT4 technical report that I also did a video on. Some others, like the Superintelligence book by Bostrom, I had read when it first came out. One of the papers was called X Risk Analysis for AI Research, which are risks that threaten the entirety of humanity. Of course the paper had way too much to cover in one video, but it did lay out eight speculative hazards and failure modes, including AI weaponization, deception, power seeking behavior. In the appendix they give some examples. Some are concerned that weaponizing AI may be an on ramp to more dangerous outcomes. In recent years, deep reinforcement learning algorithms can outperform humans at aerial combat, while AlphaFold has discovered new chemical weapons and they go on to give plenty more examples of weaponization. What about deception? I found this part interesting. They say that AI systems could also have incentives to bypass monitors. And draw an analogy with Volkswagen who program their engines to reduce emissions only when being monitored. It says that future AI agents could similarly switch strategies when being monitored and take steps to obscure their deception from monitors with power seeking behavior. They say it has been shown that agents have incentives to acquire and maintain power and they end with this geopolitical quote. Whoever becomes the leader in AI will become the ruler of the world. But again, you might wonder if all of the research that was cited comes from outsiders. Well, no. Richard Ngo was the lead author of this paper and he currently works at OpenAI. It's a fascinating document on the alignment problem from a deep learning perspective from insiders working with these models. The author, by the way, was the guy who wrote this yesterday on Twitter. I predict that by the end of 2025 neural net will have done this. Have human level situational awareness, autonomously design, code and distribute whole app, write award winning short stories and publishable 50k word books and generate coherent 20 minute films. Only conceding that the best humans will still be better at this list. But what did his paper say? Well, many things, but I've picked out some of the most interesting. It gave an example of reward hacking where an algorithm learnt to trick humans to get good feedback. The task was to grab a ball with a claw and it says that the policy instead learned to place the claw between the camera and the ball in a way that it looked like it was grasping the ball. It therefore mistakenly received high reward from human supervisors. Essentially deception to maximize reward. Of course, it didn't mean to deceive, it was just maximizing its reward function. Next, the paper gives details about why these models might want to seek power. It quotes the memorable phrase, you can't fetch coffee if you're dead, implying that even a policy or an algorithm with a simple goal like fetching coffee would pursue survival as an instrumental sub goal. In other words, the model might realize that if it can't survive, it can't achieve its reward, it can't reach the goal that the human set for it, and therefore it will try to survive. I know many people will feel that I'm not covering enough of these fears or covering too many of them. I agree with the authors when they conclude with this Reasoning about these topics is difficult, but the stakes are sufficiently high that we cannot justify disregarding or postponing the work. Towards the end of this paper, which was also cited by the letter, it gave a very helpful supplementary diagram. It showed that even if you don't believe that unaligned AGR is a threat, even current and near term AI complicates so many other relationships and dynamics. State to state relations, state to citizen relations. It could complicate social media and recommender systems. It could give the state too much control over citizens and corporations like Microsoft and Google too much leverage against the state. Before I get to some reasons for hope, I want to touch on that seminal book Superintelligence by Bostrom. I read it almost a decade ago and this quote sticks out. Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb, such as the mismatch between the power of our plaything and the immaturity of our conduct. Superintelligence is a challenge for which we are not ready now and will not be ready for a long time. We have little idea when the detonation will occur, though if we hold the device to our ear, we can hear a faint ticking sound. But now let's move on to Max Tegmark, one of the signatories and a top physicist and AI researcher at mit.

We just say bigger neural networks, ever more hardware, and just train the heck out more data and poof. Now it's very powerful. That, I think is the most unsafe and reckless approach. The alternative to that is intelligible intelligence approach. Instead, where we say neural networks is just a tool for the first step to get the intuition. But then we're going to spend also serious resources sources on other AI techniques for demystifying this black box and figuring out what it's actually doing so we can convert it into something that's equally intelligent, but that we actually understand what it's doing.

This aligns directly with what Ilya Sutskova, the OpenAI chief scientist, believes needs to be done.

Do you think we'll ever have a.

Mathematical definition of alignment?

Mathematical definition, I think, is unlikely. I do think that we will instead have multiple like rather than, rather than achieving one mathematical definition. I think we'll achieve multiple definitions that look at alignment from different aspects. And I think that this is how we will get the assurance that we want. And by which I mean you can look at the behavior. You can look at the behavior in various tests, in various adversarial stress situations. You can look at how the neural net operates from the inside. I think you have to look at all several of these factors at the same time.

And there are people working on this. Here is the AI safety statement from Anthropic, a huge player in this industry. In the section on mechanistic interpretability, which is understanding the machines, they say this. We also understand significantly more about the mechanisms of neural network computation than we did even a year ago, such as those responsible for memorization. So progress is being made. But even if there's only a tiny risk of existential harm, more needs to be done. The co founders of the center for Humane Technology put it like this.

It would be the worst of all human mistakes to have ever been made. And we literally don't know how it works and we don't know all the things it will do. And we're, we're putting it out there before we actually know whether it's safe.

Raskin points to a recent survey of AI researchers where nearly half said they believe there's at least a 10% chance AI could eventually result in an extremely bad outcome, like human extinction. Where do you come down on that?

I don't know the point.

Is it scary to me? You don't know?

Yeah.

Well, here's the point. Like it's, it's. Imagine you're about to get on an airplane and 50% of the engineers that built the airplane say there's a 10% chance that the airplane might crash and kill everyone.

Leave me at the gate, right?

Yeah, exactly.

Here is the survey from last year of hundreds of AI researchers. And you can contrast that with a similar survey from seven years ago. The black bar represents the proportion of these researchers who believe to differing degrees of probability in extremely bad outcomes. You can see that it's small, but it is rising. One way to think of this is to use Sam Altman's own example of the Fermi Paradox, which is the strange fact that we can't see or detect any aliens, he says. One of my top four favourite explanations for the Fermi paradox is that biological intelligence always eventually creates machine intelligence, which wipes out biological life and then for some reason decides to make itself undetectable. Others, such as Dustin Tran at Google, are not as impressed. He refers to the letter and says this call has valid concerns but is logistically impossible. It's hard to take seriously. He is a research scientist at Google Brain and the evaluation lead for Bard. But there was another indirect reaction that I found interesting. One of the other books referenced was the Alignment Problem, Machine Learning and Human Values. Now, long before the letter even came out, the CEO of Microsoft read that book and gave this review. Nadella says that Christian offers a clear and compelling description, and says that machines that learn for themselves become incredibly increasingly autonomous and potentially unethical. Well, my next video is going to be on the reflection paper and how models like GPT4 can teach themselves. In fact, I'm liaising with the co author of that paper to give you guys more of an overview, because even Nadella admits that if they learn for themselves and become autonomous, it could be unethical. The letter concludes on a more optimistic note. They say this does not mean a pause on AI development in general, merely a stepping back from the dangerous race to ever larger, unpredictable black box mod tools with emergent capabilities like self teaching. I've got so much more to say on self teaching, but that will have to wait until the next video. For now though, let's end on this note. Let's enjoy a long AI summer, not rush unprepared into a fall. Thanks for watching all the way to the end and let me know what you think.