Achieving provably beneficial, human-compatible AI

I don't think deep learning evolves into AGI. So AGI artificial general intelligence is not going to be reached by just having bigger deep learning networks and more data. AGI and human intelligence require fundamental capabilities that are just not present in deep learning technology as we currently understand it. Deep learning systems don't know anything, they can't reason, they can't accumulate knowledge, they can't apply what they've learned in one context to solve problems in another context, existing, etc. Etc. Right, and these are just elementary things that humans do all the time.

Bandwidth for changelog is provided by Fastly learn more at fastly.com we move fast and fix things here at Change Law because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by DigitalOcean. Digital Ocean's developer Cloud makes it simple to launch in the cloud and scale up as they have an intuitive control panel, predictable pricing, team accounts, worldwide availability. With a 99.99 uptime SLA and 24 7, 365 world class support to back that up. Digital Ocean makes it easy to deploy, scale, store, secure and monitor your cloud environments. Head to Do Co Changelog to get started with a $100 credit again. Doco Changelog.

Foreign.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changeaw.com community and follow us on Twitter. We're racticalaifn okay, take it away Chris.

Welcome to another episode of the Practical AI Podcast. My name is Chris Benson. I'm a Principal AI Strategist at Lockheed Martin. Normally listeners would know that Daniel Whitenack, my co host, would be with me. He is unavailable today. He's out sick. And so I have the pleasure of introducing our guest today who is a legend in the AI field. With me today is Stuart Russell who is a professor of Computer Science at University of California, Berkeley and holder of the Smith is it Zadeh Chair. If I'm getting that correct.

Smith, Zadeh.

Zadeh. I apologize. And also if the name sounds familiar, he is the author of the standard book on artificial intelligence which most practitioners in the field will be familiar with, as well as a recent book for a general audience which is called Human Artificial Intelligence and the Problem of Control. Stuart, thank you very much for coming on the show.

Pleasure.

I know, I barely touched on it. I know you have been in this field for decades. If you could tell us just a little bit more about your background before we get fully launching.

Yes, as you say, it's been quite a long time. I first started doing AI when I was in high school because I got a programmable calculator and I thought I could make it really intelligent. But it turned out that it only allowed 36 keystrokes in the program. So I didn't get very far with that, with that attempt, but. And then I got to use a giant computer at Imperial College, so I wrote a chess program that was my first serious AI program. I did my PhD at Stanford. I joined Berkeley in 86. So it's 34 years teaching at Berkeley and it's been a pretty interesting time. And it's, you know, most people would say now is maybe the most exciting time to be doing AI because there's so much progress. We've been able to solve or nearly solve some of the major open problems of the field. You know, speech recognition, machine translation, certain parts of computer vision, so particularly recognizing objects and images. All of those things now work pretty well. So then we can roll out all those techniques into the real world and do cool things like driving cars and everything like that. So it's lots of fun and we're all very busy.

So I guess, you know, given that you have seen so much of the evolution of this field over time, could you talk a little bit about what the field was like when you came into it and what technologies were prevalent and tell us a bit about the evolution of the field over the years all the way into the current, you know, what's certainly taken off these last few years.

Sure, yeah. So in the early years, so I guess I would say I started probably 1975, the focus was almost exclusively on problem solving, game playing, planning, logical reasoning. So everything was deterministic. So we assumed that we could give the computer perfect knowledge of the problem, a perfectly stated goal, and then it would come up with a guaranteed solution, whether it was proving a theorem or, you know, finding a checkmate, or thought, you know, coming up with a plan to deliver a bunch of parcels to a bunch of recipients or whatever it might be. And in the 80s, we had the big expert system boom. So initially logical rule based systems, so encoding expertise in logical rules. Sometimes we now call it business rules or business intelligence. That's a phrase that they use because the term expert system fell out of favor. But in the 80s, that was a really big exciting hype bubble, just like today. Sure. And the beginnings of handling uncertainty because we wanted to, to make expert systems that did things like medical diagnosis where there are no hard and fast rules. You have to take the evidence and combine it in, you know, to get some kind of soft prediction, you might say. But that technology largely failed in the real world and complicated reasons for it, but I think basically was not doing uncertain reasoning correctly. And so as you tried to build bigger systems for real problems, the whole thing would kind of fall apart. Every time you add a new rule, the other ones would start messing things up and, you know, the interactions would cause wrong answers to come out. And companies very quickly stopped investing in this. And we had what we call the AI winter where, you know, I think my AI course was down to about 25 students in 1990. It's up at about 900 right now.

I imagine it is.

And that's only because we're not, you know, the fire marshal won't let us have any more people in the class. It would probably be 12, 1500 if we let everyone in. So what happened next in AI actually was that rigorous probabilistic methods took over within the field, mainly from the work of Utopearl starting in the mid-80s. And then machine learning had a renaissance, reinforcement learning had a renaissance. And so from the late 80s until around 2011 or so, there was pretty solid technical research progress using probability statistics, connections to operations research, control theory became a very mathematical field. Some of the techniques worked pretty well. So speech recognition became reasonably practical and the first self driving cars were operating long before the present day and doing so reasonably successfully. You know, there were big applications of planning, there were lots of diagnostic systems and so on. So it was, was relatively successful. But it wasn't really until deep Learning happened around 2011, 2012 that it really hit the big time in terms of media coverage and excitement and so on. So it was deep learning that enabled us to, for example, beat the human world champion at go. Combination of deep learning with reinforcement learning methods and game playing techniques that had been around for decades and decades. But deep learning was this extra ingredient that let the system somehow recognize patterns on the GO board that allowed it to beat the human world champion. So now we're in a position where, as I mentioned, thousands of students want to take AI courses. There are thousands of startup companies, all the big companies have major AI divisions and are using AI in hundreds of applications throughout their businesses. So it's fun, it's exciting. Maybe there's some hype going on too.

There Might be a little bit. So going through that, it really begs the question for me. And I'm going to throw the question at you. I imagine we'll be like, but you're looking at all these different types of artificial intelligence, different things that are being labeled as artificial intelligence, you know, from the symbolic logic days, through expert systems, all the way up through today's deep learning and other associated technologies. And there are obviously vastly different underpinnings in terms of what they can do and how you arrive at them. How do you define artificial intelligence? What is the term mean to you as the person who has literally written the textbook on the subject and how has that changed and evolved over those years?

So it's a good question. I mean, all of the above, all the things you mentioned, this is all artificial intelligence because it's all in the service of creating machines that can act intelligently, which means really choosing actions that can be expected to achieve their objectives. If you're a self driving car, the objective is to get to the airport safely, legally, comfortably. In order to do that you need perception, but you also need symbolic planning to choose a route. You need parabolistic forecasting to deal with traffic delays and maybe have a backup route just in case. And you need speech recognition in order to interface with the passenger, etc. Etc. So you know, if you want to build a system that's going to help a mathematician, you can't just throw a bunch of theorems and proofs into a deep learning system and say here, you know, learn how to do math. You actually have to have symbolic reasoning capability, theorem proven, you know, which the underlying technology for that is symbolic logic and not statistical learning. So it all depends what you want to do. The overriding model, which I think pervades not just AI, but a lot of other disciplines, control theory, operations research, economics, statistics, they all have this model which is that we, we specify an objective and then the machine finds some optimal solution or a way of achieving the objective, the best solution. And so actually what the book is about, the human compatible book, is basically saying that model is really a terrible model. Now unfortunately, the first three editions of my textbook actually kind of solidified that model and said, okay, here's how we understand, you know, here's how we pull everything in AI together into a single conceptual framework. And you can see all of the different kinds of AI research, it's sort of different facets, different ways of looking at that same underlying conceptual framework. And the reason I think it's a terrible model is not a new Thing, right? It's something we've known for thousands of years, which is we cannot specify our objectives completely incorrectly. And, you know, if you look at the legend of King Midas, he specifies his objective. I want everything I touch to turn to gold, you know, the gods, or you could say the AI system gives him the objective exactly as he specifies. And then, of course, his food and his drink and his family all turn to gold and he dies. And misery, starvation, unexpected consequences, right? That's the thing. It's always unanticipated consequences, accidental side effects, collateral damage externalities is what the economists call it. But it's a pervasive problem. We've known about it for a long time. That's why your third wish is always please undo the first two wishes because I've ruined everything, right? So the human compatible book basically says, okay, we have to throw away that model because, you know, up to now it hasn't been that bad. Because, you know, first of all, most of our AI systems were toys. They were in the lab, you know, we were doing demos.

It wasn't out there in industry at the level it's at now until recently.

On a global scale, but now it is, right? So now, you know, you've got the content selection algorithms from all the different social media platforms. And those algorithms are. They're machine learning algorithms, but they decide what billions of people spend hours every day reading and watching. Right? So in terms of their actual direct, in terms of, like, you take the number of people times the amount of time, they are more powerful than anything that's ever existed in the history of the human race. By far. By far. Right. I mean, you know, you think Stalin was powerful, but he got to speak to his people like, you know, maybe half an hour every month or something, right? Yes. These algorithms are speaking to 50 times more people for hours every day.

A largely oblivious audience, for the most part, that's acting on them.

Yeah. So the audience doesn't know what the algorithms are doing or what they're trying to do. The algorithms are trying to maximize their objective, which is click through or engagement or something like that. And in the course of doing that, rather than just send you what's interesting, they actually modify you into someone who's more predictable from their point of view, because the more predictable you are, the more money they can make off you. And so whatever you start out as, they change you and mold you into a predictable clicker. And so that's what they've done. And I think most people would say that the results have been pretty disastrous on the whole.

So I want to ask you. There's a particular remark you make in the book that I want to ask you. And I think you're already kind of going down the path on this to some degree. But you say we must plan for the possibility that machines will far exceed the human capacity for decision making in the real world. And I think that you've started to address some of the challenges. Could you give us a little bit more of a holistic perspective on what? I mean, that statement has a lot in it right there. Can you talk a little bit about what the implications of that is?

So, yeah, so let's give an example, right? Suppose that a few years or decades down the line, you're the CEO of an IT company or your solar power company or whatever it might be, and you want your company to be more successful. So you engage an AI system and you give it the objective of, let's say, maximizing the profits or the revenues of my corporation, because that system is far more capable than humans are. It devises plans that are more successful than all the competitors can be. And so that corporation, in the interests of maximizing revenues, gradually takes over larger and larger portions of the world economy. And if it's not properly designed, if that was the objective, wherever it was feasible, it might end up using slave labor, for example, in order to maximize profits and so on and so forth. You can imagine all the ways that corporations have abused humanity in the past, and now we've got one that's much more capable than human beings are. Some people actually argue that this is already happening not from AI, but from corporations that optimize profit at the expense of everything else. So, for example, at the expense of the climate, the fossil fuel industry has optimized its profits by sort of multi decade misinformation strategy that's actually outwitted the human race, right? And so even though the vast majority of experts and economists and scientists say, oh, you know, we need to have a carbon tax, we need to do this, we need to do that, we aren't doing any of it, right? We're just talking about it. And so effectively, the fossil fuel industry has defeated the human race by superior pursuit of a fixed objective. So it would get much worse than that when AI systems are able to invent and carry out these kinds of strategies. So that's even, you know, within the realm of things that we currently understand, right, that you could have corporate strategy, you could enslave people, you could do this, you could do that. But AI systems will come up with things we don't understand.

Sure.

And we, you know, the whole human race could be collateral damage if we don't know how to control the systems that we create. And so far there's no examples of a dumb species controlling a more intelligent species forever.

I totally agree with that. So for my own employer, I'm actually the person leading on AI ethics. So AI ethics is a huge passion of mine, and obviously you've raised some pretty big concerns there. And I'm taking a little bit of a tangent. I wasn't really expecting to go down this path. But I am curious how you envision the role of AI ethics in our society, in the world at large, given everything that you just said. I mean, clearly the potential for consequences that we did not envision, that we did not plan on is fairly significant, especially as technology evolves. Do you have any thought.

In a sense, I wish it wasn't called AI ethics.

Okay, what should it be called?

Well, so, I mean, let's give you analogy. Right. So the nuclear engineers who make sure that nuclear power stations don't explode like Chernobyl, are they ethicists? Would you say that's a nuclear ethics issue? No. I mean, it's just common sense that you don't want your nuclear power station to explode. It's common sense that you want your AI systems to remain under human control.

Sure.

But at the moment, under the Standard Model, they won't remain under human control.

And would you talk us through what that implies when you say it won't? And I'm going to set it up in this way. And that recognizing. And it's funny how many people I talk to have different perspectives from what I think you're about to go. But given the evolution that we've seen over time and the rapid evolution we're seeing in deep learning and whatever follows coming up, that potential for loss of human control, where does that come from? Why is it inevitable, in your view?

I don't want to say inevitable if we persist with AI within the Standard Model where we fix an objective. Because when you fix an objective, you're basically telling the system whatever course of action optimizes that objective is the correct thing to do. In particular, for example, anything that imperils the success of the objective has to be prevented. Well, what might imperil the success of the objective? Well, being switched off.

Sure.

So by giving a system a fixed objective, you've now given it an incentive to protect itself from any attempt to interfere with the objective, from any attempt to switch it off.

So as a very typical argument I hear people make, if you kind of go back to like, you know, Asimov's three rules for robotics, you know, in fund in the idea that you can just in a non probabilistic way just definitively say, you know, you can't hurt people, that kind of thing as an underlying thing. Given the fact that you have this ever increasing capability in the AI realm, would it be fair to say that's not a realistic perspective that the intelligence that AI would fundamentally look to circumvent or how do you see that?

Yeah. So Asimov's Law is as you say, don't take into account the probabilistic perspective. They don't allow for uncertainty. But of course in the real world there are always risks. So you know, an Asimovian self driving car would simply stay in the garage. It would say, I'm sorry, you know, the first floor does not allow me to leave the garage because that would expose you to risk of injury or death. So sorry, we're not going anywhere.

I love that. That's very funny actually.

It's true. And if you were out for a walk, it would run around with an umbrella. In case of photon from the sun, you know, landed on your skin and maybe initiated little melanoma or something like that. There's a chance that could happen. So we have to protect it. So you know, in any kind of real world situation there are trade offs. But one of the things that Asimov's Laws do is they make a start on saying what it is that humans want. Right? One of the things, we don't want to be harmed, we don't want to be physically injured. And that's a start. Because you know, for example, none of the self driving cars that are out there right now know that people don't want to be injured.

Understood, Right.

They have built in rules that say, well, if there's a pedestrian in front of you and you're going forward, stop, right? And if there's a, if you're lucky they have another rule that says if there's a pedestrian behind you and you're going backwards, stop. Right. But they don't know why. They don't know that if you run into a person, you can injure or kill them. And they don't know that the person doesn't want to be injured or killed. And it's that lack of knowledge actually that makes them very brittle because when they get into situations they haven't been prepared for, they haven't the Faintest idea what to do because they don't know which course of action is good and which is bad. So the solution that the book proposes actually is to say, look, it doesn't matter how much the human tells you that they want this or they don't want that. There's always going to be residual uncertainty about other preferences the human may have. So if the human says, I'd like a cup of coffee, that's not your life's mission. Right. You know, the robot could say, well, you know, the coffee in this hotel is 15 bucks a cup. Are you sure you want a cup of coffee? Right. Because this machine is uncertain about your trade off between coffee and money. Right. If you're miles from the nearest coffee, the robot might not be sure. Do you want to wait two hours for this coffee? Is it okay if I trundle off across the desert to the nearest Starbucks and come back two hours later or two weeks later with a cup of coffee? So it would be reasonable again to ask permission and you know, if you give it a more important goal, like restoring carbon dioxide levels to pre industrial concentrations, if that was the only objective. Well, you know, one very straightforward solution is just to get rid of all the people understood, because they are the ones who are producing the carbon dioxide. And then you might say, oh, well, I didn't, I didn't mean that. All right, so wish number two, restore the carbon dioxide, but don't kill anybody. And then the system says, fine, fine, no problem. We'll just have a multi decade social media campaign to convince people not to have children. And then the human race will gradually die out and then carbon dioxide levels will be restored. And that's great. So what I'm really proposing in the book is actually throw away the standard model, or only use a standard model in very restricted circumstances, but in general have a new model where the objectives are in the human. And the machine knows that it doesn't know what they are. Its job is to try to satisfy them, but it knows that it doesn't know what they are. And when you design things that way and you actually solve that problem. So you, you can have an algorithm that for that problem specification decides what the machine is going to do. That algorithm produces behaviors that seem to be what we want, namely asking permission. Like, you know, is it okay if I turn the oceans into sulfuric acid in order to restore carbon dioxide levels? And you say, no, not so much. We like those little fishies don't turn the oceans into sulfuric acid. Right. So we will ask permission, you know, it'll even allow itself to be switched off. So rather than try to protect itself and take steps to prevent interference, it actually welcomes interference because interference by a human is a way of gaining information.

What up nerds? I got some pretty awesome news to share with you. Pluralsight is totally free for the entire month of April. I'm not kidding. Seriously. Head to pluralsight.com changelog and skill up while you stay at home for the entire month of April, you'll get access to over 7,000 courses from experts in software development, security, cloud and data. There's never been a better time to skill up. Head to pluralsight.com changelog again pluralsight.com changelOG.

So if you would take us a little bit farther into your the the new model that you're proposing to replace the standard model, and maybe along the way, one of the things I was wanting to ask as you were discussing that is if you could do that also in the context of as we're looking at AI in the deep learning context of today, anticipating wherever we may be going in the future and, and with the idea that people talk about about AGI, which is artificial general intelligence, which presumably would change the nature of, of what AI is maybe and maybe distinguish how your new proposal would work kind of in both worlds. I mean, if you were even today, as we're looking at exceeding human capability by if you have a complex set of tasks even now, we can take the models that we have and have many models, each one addressing kind of a narrow scope and working together and they can far outperform what humans could do in a similar complex task. And with the idea also of having AGI, where we have models that are, for lack of a better word, more capable in themselves, maybe eventually aware. I don't know if you could kind of talk about what your proposal looks like in that evolving world. I'd love to know.

Sure. So first of all, I should point out that I don't think deep learning evolves into AGI.

Okay.

Right. So AGI, artificial general intelligence is not going to be reached by just having bigger deep learning networks and more data. AGI and human intelligence require fundamental capabilities that are just not present in deep learning technology as we currently understand it. So deep learning systems don't know anything. They can't reason and they can't accumulate knowledge. They can't apply what they learned in one context to solve problems in another context, etc, etc, and these are just elementary things that humans do all the time.

A bit of a stepping stone technology of the moment, in a sense, the deep learning.

Well, I think deep learning is one of the pieces, but you know, so is symbolic logic, so is probabilistic reasoning, so is sequential decision making, techniques, planning, hierarchical reinforcement learning, probabilistic programming, etc, etc, so there are, you know, there's lots of pieces of the puzzle, some of which have been lying around for a long time. Deep learning is just the newest, shiniest one. So everyone's sort of, oh, look, you know, but you know, in the 80s people were going, oh look, expert systems, you and similar claims were being made, right, that we just, you know, if you just like scale up the number of rules by a factor of 500, you know, and you had like learned people, you know, making quantitative estimates, like, oh yeah, we would need about 500,000 rules to manage your military campaign and stuff like this, like just complete drivel. Yeah, and there's a lot of drivel being talked now about deep learning. But okay, so within the context of just straightforward supervised learning, let's say for image classification. Okay, Right. So how does it work? Well, we have training data and then we have deep learning, which is basically a giant tunable circuit with billions of tunable connection strengths, like little tiny volume controls. And we just tweak all those volume controls in this huge circuit until the thing that comes out the other end is the correct classification of an image. So in statistical learning, what you do is you, you're supposed to specify the loss function which says if you classify an object of type A as an object of type B. So let's say you class picture of a dog and you classify as a cat, how bad is that? Right, so almost everybody in this business uses what we call a uniform loss function, which means that they say every error is equally bad because that's how the competitions work. They penalize you for the number of errors you make, not how bad the errors are. Right. So, you know, for example, in Imagenet, there are two categories of dog. One, well, there's 100 and something categories, but two of them are the Norfolk terrier and the Norwich terrier. Right. And these are practically identical. In fact, they weren't even recognized as separate breeds of dog until 1960 something, you know, and there's like slight difference in the shape of the ear and it's like, okay, who cares? I'm sure the Norfolk Terriers are not going to be that upset if you call them Norwich terriers. I mean, Norwich is in Norfolk.

Anyway, they'll lick your face either way. Right?

So clearly that kind of error is relatively cheap, whereas classifying a human as a gorilla, as Google found out, is really expensive, like in the billions of dollars of, you know, trashing your goodwill of your corporation and its global reputation for being fair and idealistic and all the rest of it. Right. You know, it was, yeah, I'm sure it was sort of an innocent error coming from just using a uniform loss function.

Sure.

And, but if they had thought about it, they would have said, oh, of course our loss function is not uniform. Oh, then what is it? Oh, dunno, we haven't thought about it and we're not sure. And in fact, you know, if you've got typically an Imagenet, there's like 20 something thousand different categories of object. So your loss function is a matrix with 400 and odd million entries. And do you know what they are? No, no one knows what they are. So you have an uncertain objective. You don't know what the objective is you're supposed to be optimizing. And when you formulate the problem that way, first of all you'd have to say, okay, well, how do we specify a probability distribution over these 400 million entry matrices, these giant tables? And now I got to say, okay, what's the probability of each possible table of 400 million numbers? Well, that probability distribution is itself massively complex object to specify. And no one has ever figured out even how to write it down, how to structure that probability distribution because it clearly it has lots of structure. Right. The costs of misclassifying each breed of dog as a cat is probably about the same. I think all dogs are equally upset to be called cats. But if, you know, if you classify a bus as an insect, maybe that's a more embarrassing mistake to make, and so on. So you can imagine that there's lots of structure in this, in this matrix. And you know, the structure partly reflects a taxonomic hierarchy of objects and how we arrange them into categories. So you could do a whole PhD thesis just on that part of the problem. And now there's also, well, what does the algorithm look like? Right? Well, if it doesn't know the loss function and it has the opportunity to find out more, for example, by asking the user, you know, is it worse to call a cat a dog or to call an apple an orange? And sometimes the algorithm would say, I'm not going to classify that image, right, that's too dangerous, so I'm just not going to make a guess as to what it is. So you immediately see that just the nature of supervised learning would change considerably if you allow for uncertainty about the underlying objective. And then with AGI, we don't yet know exactly how to build AGI. I mean, there are a bunch of unsolved major conceptual problems that we have to figure out. But I think the basic answer is that if you formulate AGI within this new model, the key property of the new model is that the better the AI solves the problem, the better the outcome is for human beings, because it means that the AI system does a better job of finding out what it is you want and does a better job of achieving it.

And so you were talking a moment ago about kind of applying control in the new model that you're proposing. As we move into AGI, would you pick that train of thought right up where you left it there?

Yeah. So with AGI, if we formulate it in the new model, the. The key property is that the smarter the AI, the better the outcome for humans, because the AI system will be able to better interpret our behavior as evidence of our underlying preferences. So this is the nature of information flow between the human and the machine about what the human's objectives and preferences are, is that everything the human does reveals evidence for our underlying preferences. So the AI system, observing us, observing our whole history, observing everything we've ever written, is able to infer from that something about what we want as individuals, as a species, and so on. And so the better the AI system, the better job it'll do of that, and the better it'll be able to achieve those objectives.

When you say that, just to clarify, it sounds like you're going into unsupervised learning, where it just has kind of the wealth of. Of human knowledge and what humans have done? Or are you strictly leaving it for. In this new approach, Are you leaving it for the algorithm that you're training to figure that out? Are you specifying it as the practitioner? Do you see this as kind of at some point, maybe leaving kind of today's deep learning behind and taking a different approach mathematically? How does that look going into the future if everyone adopts this?

Well, yeah. So you often see this claim that there's supervised learning, unsupervised learning, and then. Well, that. I mean, logically, if those were correct, then supervised and unsupervised with, that constitutes a complete coverage of all learning, right? A and not a but. And then they say, oh, and there's reinforcement learning as well, but actually there's not. There's other kinds of learning too. And this is related to something we call inverse reinforcement learning and inverse reinforcement Learning is basically. Well, so let me first say what reinforcement learning is. Reinforcement learning is the human specifies the reward to the machine, and then the machine learns how to optimally produce reward. So the machine says, okay, this is I'm going to give you one point when you win the game and I'm going to give you zero when you lose. And then the machine learns to get one point more often than not. So inverse reinforcement learning is the other way around. The machine is observing, let's say, the human and trying to figure out what is the reward function that this human is optimizing. And we came up with it actually when I was collaborating with some biologists and we were trying to figure out how could we apply reinforcement learning to understand animal locomotion. So, you know, cockroaches and flies and other, you know, creepy crawlies and so on. And it struck me that actually we can't apply reinforcement learning to sort of simulate animal, you know, to create a simulated insect or whatever, because we don't know what the reward function is. Right, right. So then I say, oh, well, why don't we watch them walking and figure out what reward function are they optimizing, you know, with their particular choice of how to locomote. Right. Because, I mean, I don't know if you've ever seen the Monty Python silly walk sketch.

Yes, I have.

Your listeners may want to check that out on the web. Right. So John Cleese demonstrates that there are many other ways you could walk besides the usual one. Right. So we choose the usual way of walking because it does something good for us, whether it's energy efficient or stable. You know, it avoids falling over, whatever it might be. It's optimizing something. And so the idea of inverse reinforcement learning is observe the behavior and figure out what is being optimized by this behavior.

Okay.

And so this, this approach, the new model, is a sort of a generalization of that idea because it's generalized in the sense that the human is not just being passively observed behavior, doing whatever human thing, but the human is sort of an active participant. For example, if the human solves their half of this problem, they will actively teach the robot about their preferences, including saying things like, I would like a cup of coffee. Right. That's conveying preference information to the robot. It's not an order, it's sort of just factual evidence about my state of mind. And the robot can interpret it as it wishes. So when you, when you solve this kind of problem, it's what the economists call a game, which means A decision problem with more than one decision making entity. So you can imagine one human and one machine, or lots of humans, lots of machines. So you can solve that problem mathematically and then you just look at the behaviors that the machine and the human engage in when they solve this problem. And indeed the human teaches the machine and the machine does things, it asks permission, it allows itself to be switched off, and so on. So you get very different behaviors than you do in the standard model of AI. And so I think I'm reasonably optimistic that in fact it shouldn't matter how intelligent the AI system is, things will still go well. And in the old model, the more intelligent the machine, the worse the outcome for people because the machine would find some way of messing with the world to achieve the objective that you said and mess with the things you forgot to mention that you care about.

So I'm kind of curious as you take this and you're looking, we've, we've hit so many different areas and so I'm, I'm trying to tie it together. You look into the future at this point, having come as far through this field as it's evolved and changed over the years, where do you see it going, especially with control in mind and, and you know, as you've talked about how the current standard model can lead us awry, then if you are a practitioner and you're out in industry and you're trying to do the things that your organization wants you to do, how do you apply your new model? And as you look out, what do you think we're going to be doing in terms of what types of models, what is AI kind of evolving into? If you're looking out five years or 10 years and we're learning these lessons that you're teaching us in this capacity, what does the near mid and a little bit farther out look to you at this point?

Interesting. So first of all, there's going to be a little bit of pushing and shoving, right? I would imagine the AI community that's grown up with the standard model and learned it from the textbook is going to keep pushing ahead with their solving the technical problems that they're solving within the standard model and they have to be dragged kicking and screaming into this new way of doing things. So partly means we, you know, my research group, there are maybe a dozen other research groups around the world working in this new framework now. We have to provide the technical solutions, we have to provide the new algorithms that behave according to the different principles. And I think we can do that if we can do that in practical settings, whether it's, you know, recommendation systems, content selection for social media, intelligent personal assistance, then I think that will have a significant effect. People will say, oh, now I get it. In fact, no, they won't say, now I get it. They'll say, oh, I always thought that way, of course. So, you know, there won't be a sort of, you know, a moment of capitulation. There'll be a. Just a gradual realization that, of course, this is what they've always thought. And it makes sense, right?

It does make sense. Do you think that as they adopt this, as I'm thinking about what you're saying here, and you already mentioned that the poorly named idea of AI ethics in terms of how do we prevent those, how does that come together? Because there's the algorithmic side, there's the new algorithms where you are going out into the future and you're implementing inverse reinforcement learning and it's working for you technically. And you're also trying to say, we want to ensure that the outcomes are beneficial and certainly to the human involved. How does all that come together? Because right now, you know, as I look at people, there are people that are doing kind of the. The outcome, the ethical concern. There are. There are people that are strictly algorithmically focused in terms of solving problems. And yet, if I'm understanding you correctly, you need to be able to fuse all these together. It sounds like. So that that works.

Yeah. Because I think the last thing you want, and you've probably experienced this yourself, is that AI ethics people leaning over the shoulder of the AI practitioners and wagging their finger and saying, no, no, you're a bad person. Right. Doesn't work. So what we have to do, actually, is to get people to understand that this is just how you do AI. When civil engineers design bridges, there's not a bunch of bridge designers and then a bunch of ethicists saying, oh, by the way, you have to make sure it doesn't fall over. Right. It's just, of course, of course it's not supposed to fall over. It wouldn't be a bridge if it just fell over. And similarly, nuclear engineers who design nuclear power stations, there isn't, you know, another discipline of people who care about safety. And then the nuclear engineers don't care about safety and they just want to generate lots of energy. Right. Doesn't work that way. If we want to generate energy, we can just set off lots of bombs. That's cheap and cheerful. Gets past all this red tape and all that crap. So I think There should be a strong incentive to just design systems this way because they won't fall down. Right. The example I use in a lot of talks is that your domestic robot, if it's designed this way, won't cook the cat for dinner. Because there is, you know, because the fridge is empty. Because it would say, well, I mean, it might realize, yeah, cooking the cat for dinner solves the problem of lack of food. But I'm uncertain about, perhaps the cat has sentimental value and so I shouldn't cook it. I should ask permission before I cook it. Right. So you get better behavior out of your AI systems. And so they'll be economically more valuable. They'll incur far less in the way of liability insurance and so on. So there's that. But I also think that we at some point will need regulation because there will always be, just as there is with malware. There's a tendency, you know, temptation to just bypass safety and all the rest just in terms of immediate grasping. And so as AI systems become more and more capable and potentially powerful, they need to be regulated more and more strictly, just as we do with nuclear power stations.

Sure. And I would imagine that's not just at a national level, but they'll have to be a body of international law, because you have different parts of the world, different countries have different values that they're bringing to play, and some are going to care more about these kind of outcomes than others. I guess I wanted to finish up with, you know, you have all these students looking to you to. And they're coming in and they are starting their careers out in this field. You have people like me who are a little bit older, and we are, we are trying to constantly retool and stay up with the field. And you've really, you've kind of shifted, you know, in this conversation that we've just had, you've really shifted how I'm looking at the future and the things that need. How does a practitioner or a student that's about to be a practitioner kind of tool themselves today beyond just the current state of deep learning? Because that's where all the focus is right now. It's all about TensorFlow or pick whatever tool you want to use. And we're building neural networks or adjacent technologies, and that's where all the education is really focusing. That's broadly available out there on the Internet and by service providers and others. How should someone like myself or a student coming in the field be thinking about this? And how should we focus on educating ourselves for the future to Align ourselves with this vision that you just set out. Obviously, there's your book, there's Human Compatible. I don't know if it's out yet. The fourth edition to your textbook.

Yeah. So the fourth edition is out next week. We finished it a few weeks ago and I think it'll be in the stores in a week's time.

Perfect timing. As they hear this, they can go out and buy it. Yeah.

So, you know, it's an unfortunate situation because we basically, we've put the technical content from Human Compatible into the new edition of the textbook, you know, so the first two chapters saying, okay, well, there's this old way of thinking about AI and now there's this new way. But we don't have all the chapters in the middle telling you how to do the new way. So we're going to tell you how to do the old way. But keep in mind that you really should be thinking about the new way.

So that'll have to be the fifth edition.

An awkward timing. Yeah. So the fifth edition will hopefully have more stuff. But the things to keep in mind are, first of all is the objective that you're designing your system to optimize. And I think, as I mentioned, with the example of image classification and classifying the person as a gorilla, most people are not even thinking about that. The objective there is typically implicit. When you run TensorFlow, if you don't put in the loss function, then you're putting in a uniform loss function. And if you put in a uniform loss function, you're saying that classifying a human and gorilla is just as bad as classifying one kind of terrier as another. Right. And that's not true. So don't do it. Right. The second thing to think about is what is the scope of action of your system? So if your system could learn any function that strings together actions that it carries out, what is the sort of transitive closure? What's the full set of states that your system could take the world into when it runs in the real world? And if, for example, you're just writing a Go program and it's only moving pieces on a simulated board within the memory of the computer, and then displaying it's relatively safe because no matter what sequence of moves it does, it's still only changing what appears on the screen when someone's playing go with it. Now, theoretically, it's not perfectly safe because just as we have learned the origins of our own universe and the physics of the world in which our bodies run, a sufficiently intelligent Go program could actually do the same thing and then figure out that there must be other entities outside of its computer and try to communicate with them through the go board and convince them to give it more CPU power or whatever. Right. So it's not hermetically sealed even then. But if your algorithm is in direct contact with humans. Right. Then here's one good way to remember this. Hitler did it with words. Hitler was not 1000 foot tall giant robot with laser beam sweeping destruction everywhere. It was just a little guy who spoke.

Yeah, that's a great point.

And so if your AI system is in direct contact with humans, it has far more power than Hitler already because it can speak to billions of people all time.

That may be the most terrifying thing I've ever heard a person say. Right there. That's perspective right there.

Yeah. If you think, you know, what is the closure? Right. What is the transitive closure of all possible actions that system could do? There's really no limit to it.

Yes.

Right. It could affect the world in any way. So those kinds of systems, I think, should absolutely be carefully regulated. And I think, for example, we talked earlier about how social media algorithms work. I think you can distinguish between reinforcement learning algorithms in that context and supervised learning algorithms. So a supervised learning algorithm, roughly speaking, will learn what people want, whereas a reinforcement learning algorithm will learn to manipulate people to change what they want so that the algorithm can make more money off. And so, you know, I really think that algorithms that are facing the general public in that way need to be regulated not in exactly the same way, but in some way analogous to the way we regulate pharmaceuticals. We don't just get to spit out, you know, new pharmaceuticals to billions of people whenever we feel like it. Right. They have to be carefully tested on, you know, on animals and then on, you know, control groups of humans, because if it goes wrong, it's really bad. And the same as we learned is true with these social media algorithms.

Well, I just wanted to say thank you so much for coming on the show. This has truly been one of the most fascinating conversations I've ever had. I think at this point I will be recommending that people read human compatible pretty much everywhere I go.

Well, thank you.

You point out the danger of as we grow here if we don't start taking that into account. So thank you so much for coming on the show. You've really blown my mind. So I don't normally finish the show stuttering like this. So thank you very much.

Thank you, Chris. It's nice talking to you.

Thank you for listening to this. Episode of Practical AI more like this@changelog.com PracticalAI there you'll find our latest as well as lists of our most popular episodes and the ones we recommend. Recommend if this show has helped you on your AI journey, please leave us a five star review on Apple Podcasts, heart us on Spotify, star us on Overcast, and tell a friend what they're missing out on. Practical AI is hosted by Daniel Whiteneck and Chris Benson. It's produced by me, Jared Santo and our music is brought to you by the Beat Freak Breakmaster Cylinder. We have awesome sponsors. Please support them. They support us. Thanks again to Fastly, Linode and Rollbar. That's all for now. We'll talk to you next time. Time.