Stuart Russell: The Control Problem of Super-Intelligent AI | AI Podcast Clips

So let's just talk about maybe the control problem. So this idea of losing ability to control the behavior of AI system. So how do you see that? How do you see that coming about? What do you think we can do to manage it?

Well, so it doesn't take a genius to realize that if you make something that's smarter than you, you might have a problem. Alan Turing wrote about this and gave lectures about this. I think 1951, he did a lecture on the radio and he basically says once the machine thinking method starts very quickly they'll outstrip humanity. And if we're lucky, we might be able to. I think he says we may be able to turn off the power at strategic moments, but even so, our species would be humbled. Actually, he was wrong about that. Right. If it's a sufficiently intelligent machine, it's not going to let you switch it off. It's actually in competition with you.

So what do you think is meant just for a quick tangent, if we shut off this superintelligent machine, that our species would be humbled.

I think he means that we would realize that we are inferior, right? That we only survive by the skin of our teeth because we happened to get to the off switch. That was a close call, just in time. And if we hadn't, then we would have lost control over the Earth.

So are you more worried when you think about this stuff about super intelligent AI, or are you more worried about super powerful AI that's not aligned with our values? So the paperclip scenarios, kind of, I think so.

The main problem I'm working on is the control problem. The problem of machines pursuing objectives that are, as you say, not aligned with human objectives. And this has been the way we've thought about AI since the beginning. You build a machine for optimizing and then you put in some objective and it optimizes. We can think of this as the King Midas problem, right? Because if, you know, so King Midas put in this objective, right? Everything I touch should turn to gold and the gods, you know, that's like the machine they said, okay, done. You now have this power. And of course, his food and his drink and his family all turn to gold and then he dies. Misery and starvation. And this is, you know, it's a warning, it's a failure mode that pretty much every culture in history has had some story along the same lines. You know, there's the, the genie that gives you three wishes and, you know, third wish is always, you know, please undo the first two wishes because I messed up and, you know, and when Arthur Samuel wrote his chess, his checker playing program, which learned to play checkers considerably better than Arthur Samuel could play and actually reached a pretty decent standard. Norbert Wiener, who was one of the major mathematicians of the 20th century, sort of the father of modern automation, control systems, he saw this and he basically extrapolated, as Turing did, and said, okay, this is how we could lose control. And specifically that we have to be certain that the purpose we put into the machine is the purpose which we really desire. And the problem is we can't do that. Right.

You mean we're not. It's a very difficult to encode. To put our values on paper is really difficult, or you're just saying it's impossible.

The line is between the two. So theoretically it's possible, but in practice it's extremely unlikely that we could specify correctly in advance the full range of concerns of humanity.

You talked about cultural transmission of values, I think is how humans to human transmission of values happens, right?

Well, we learn, yeah. I mean, as we grow up, we learn about the values that matter, how things, how things should go, what is reasonable to pursue and what isn't reasonable to pursue.

I think machines can learn in the same kind of way.

Yeah. So I think that what we need to do is to get away from this idea that you build an optimizing machine and then you put the objective into it. Because if it's possible that you might put in a wrong objective, and we already know this is possible because it's happened lots of times, that means that the machine should never take an objective that's given as gospel truth, because once it takes the objective as gospel truth, then it believes that whatever actions it's taking in pursuit of that objective are the correct things to do. So you could be jumping up and down and saying, no, no, no, you're going to destroy the world. But the machine knows what the true objective is and is pursuing it. And tough luck to you. And this is not restricted to AI. This is, I think, many of the 20th century technologies. So in statistics, you minimize a loss function. The loss function is exogenously specified. In control theory, you minimize a cost function. In operations research, you maximize a reward function, and so on. So in all these disciplines, this is how we conceive of the problem. And it's the wrong problem because we cannot specify with certainty the correct objective. Right. We need uncertainty. We need the machine to be uncertain about what it is that it's supposed to be maximizing.

Favorite idea of yours? I've heard you say somewhere. Well, I shouldn't pick favorites, but it just sounds beautiful. We need to teach machines humility. Yeah, it's a beautiful way to put it.

I love it that they're humble in that humble AI, they know that they don't know what it is they're supposed to be doing and that those objectives, I mean they exist, they're within us. But we may not be able to explicate them. We may not even know how we want our future to go. Exactly. And a machine that's uncertain is going to be deferential to us. So if we say don't do that, well, now the machine's learned something a bit more about our true objectives because something that it thought was reasonable in pursuit of our objectives turns out not to be. So now it's learned something, so it's going to defer because it wants to be doing what we really want. And that point I think is absolutely central to, to solving the control problem. And it's a different kind of AI. When you take away this idea that the objective is known, then in fact a lot of the theoretical frameworks that we're so familiar with, Markov decision processes, goal based planning, standard game tree search, all of these techniques actually become inapplicable and you get a more complicated problem because. Because now the interaction with the human becomes part of the problem because the human body making choices is giving you more information about the true objective. And that information helps you achieve the objective better. And so that really means that you're mostly dealing with game theoretic problems where you've got the machine and the human and they're coupled together, rather than a machine going off by itself with a fixed objective.

Which is fascinating on the machine and the human level. That when you don't have an objective means you're together coming up with an objective. I mean, there's a lot of philosophy that you could argue that life doesn't really have meaning. We together agree on what gives it meaning and we kind of culturally create things that give. Why the heck we are on this earth anyway. We together as a society create that meaning and you have to learn that objective. And one of the biggest. I thought that's where you were going to go for a second. One of the biggest troubles we run into outside of statistics and machine learning and AI and just human civilization is when you look at I came from, I was born in the Soviet Union and the history of the 20th century, we ran into the most trouble, us humans, when there was a certainty about the objective and you do whatever it takes to achieve that objective. Whether you're talking about Germany or Communist Russia.

Yeah, I would say with corporations. In fact, some people argue that we don't have to look forward to a time when AI systems take over the world. They already have, and they're called corporations. Corporations happen to be using people as components right now, but they are effectively algorithmic machines and they're optimizing an objective, which is quarterly profit that isn't aligned with overall well being of the human race. And they are destroying the world. They are primarily responsible for our inability to tackle climate change. So I think that's one way of thinking about what's going on with corporations. But I think the point you're making is valid, that there are many systems in the real world where we've sort of prematurely fixed on the objective and then decoupled the, the machine from those that it's supposed to be serving. And I think you see this with government. Government is supposed to be a machine that serves people, but instead it tends to be taken over by people who have their own objective and use government to optimize that objective, regardless of what people want. SA.