OpenAI, reinforcement learning, robots, safety

Bandwidth for changelog is provided by Fastly. Learn more at fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@robar.com and we're hosted on Linode servers. Head to linode.com changelog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community follow us on Twitter. We're at PracticalAI FM and now onto the show.

So Woek, thanks for joining us in practical AI. I know you've got a busy schedule today at O'Reilly AI, so thanks for taking time.

Thank you for the invite.

Yeah, and you probably don't need much introduction, but for those that don't know, Wojciek is one of the co founders of OpenAI and he's going to be talking today at O'Reilly AI about robotics and deep learning. Is that right?

That's correct, yeah.

Awesome. It'd be great if you could give the audience just a little bit of an idea of your background and what you're working on now.

Cool. So I spent some time at Google Brain, I spent some time at Facebook, I researched, I finished PhD in the topic of deep learning even I spent years, many years ago sometime at Nvidia before actually deep learning was the thing. So it's kind of somewhat coincidental that all these things turn out to be extremely relevant as you have said. Daniel I'm one of the Co founders of OpenAI and Goal of OpenAI is quite ambitious is to figure out the way to build general artificial intelligence or to be more exact, how to build it in the way that say it's safe, in a sense we can control it or let's say figure out, let's say from political perspective how to deploy it in the way that it is beneficial to humanities as a whole. Our approach is more or less as follows. We see various limitations of current systems and we think what's the best way to what are the goals that we should attempt such that if we solve them it becomes clear that we lifted these limitations. And particularly, I mean there are, let's say somewhat several kind of internal approaches, how we go about the problems. But in case of what I'm doing, the motivation is as follows. So we know as of today that you can take extremely complicated computer games and machine can become superhuman in it. But the main criticism is yeah, but that's just the game. It is kind of, let's say, unrelated to reality, confined into, let's say, the realm of the computer and kind of sounds straightforward that we should be able to pull it off and apply it into real world. But turns out that many people tried and haven't succeeded for a while when it comes to very. To let's say, more complicated robots and so on. Yeah, I mean, I will let you ask me more questions and then I can tell you more.

Sounds great. Yeah, I appreciate that. I was wondering before going forward if you could just maybe. I know you mentioned, you know, OpenAI kind of going after general AI or general intelligence. I was wondering if you could kind of break that down for the listeners in terms of how, how that might be different from the research or the methods that other people are going after.

So this clear to me that AI AGI consists of more or less three components. So I mean, you have to be able to inject gigantic amount of compute. Most likely you need to invent some new algorithms and AI needs some kind of a data or environment in which it lives. So there are three main components that you have to, let's say, attack if you on just the capabilities of AGI. So I think that OpenAI is extremely well suited to pursue this goal because of few reasons. One, we are not bounded by a product. So often when you are building a product, actually quite huge fraction of the work has to do with figuring out the niche or let's say target marketing, let's say packaging and so on. And it's clear that there are various problems that you can attempt that have very small economical value. But it's actually very clear that they are making actual progress. I mean you would for instance, be able to train the system that can solve Riemann hypothesis like a very.

And could you just give a quick explanation of what that is?

Okay, yeah. So let's say I just mentioned Riemann hypothesis is like unresolved mathematical problem. So if system would be able to have such an incredible reasoning that would indicate that you lifted one more restriction. Maybe the restriction that the networks do not reason much has not that much of an economical value. I mean, maybe people doing math would get upset, you would unemploy another group of people. But in a sense that's not the product out of which you are making tons of money. So I'm just saying that. So I just compared, let's say our approach to building conventional product. I can also compare it to let's say academic labs. So in a sense most of the Academic labs, they construed of the. It's like an endeavor of small number of individuals. And I would say among these three components that I mentioned, let's say compute algorithms and environment data. It actually focuses mostly on the algorithm, which is completely fine. That's one of the components, that's the one that they are well suited to actually pursue. But it seems to me that you have to be able to pursue actually all three components to make progress. You need to be able to focus the Entire data center 10 miles by 10 miles, that consumes millions of watts with some number of algorithms and let's say environments to actually achieve it. So in essence you need resources beyond actually let's say researchers. You need also all sorts of the talented engineers, infrastructure engineers and so on and so forth. So I guess that's the main difference.

Yeah, that helps a lot. And you kind of alluded to the fact that in your background you had experience with these different technologies that kind of coincidentally converged in this new hype around AI and you co founded OpenAI. And I want to get into the content of your talk a little bit later, but I'm wondering if you could just comment on what you see as some of the advantages and disadvantages of kind of founding a company that is really at the center of this hype around AI and what your thoughts on that are and how you see the field moving forward in the midst of all this hype. I think you know Nips, the NIPS conference sold out in like 15 minutes. It's like faster than a Taylor Swift conference or something.

People are comparing the sales of tickets for NIPS to sales of tickets to birding when.

Yeah, exactly. Yeah, so and OpenAI is really at the, at the kind of center of that hype. Have you seen that kind of advantages and disadvantages to that or in terms of the progress that you're going after?

So I mean this extremely important to let's say not over promise deliver and so on. I mean otherwise over some period of time is going to bite you in the ass. Simple as that. The fact of being well recognized organization definitely helps to hire incredibly brilliant people. I would say I'm feeling pretty much like I'm feeling extremely honored that say people around, they are on various axes. Pretty much everybody is on some axis better than me. I'm feeling honored that these folks want to work with me.

Awesome. Well, yeah, thank you for that and I definitely respect a lot of the things you're doing and so looking forward to seeing what happens from here. But now I Want to kind of talk a little bit. You're going to be talking about deep reinforcement learning for robotics at OpenAI. Give us just a brief sketch of what you'll be talking about.

Cool. So as I mentioned at first, it's actually the case that reinforcement learning, it's quite generic, incredible paradigm. It's effectively saying that you can take any quantity that you can measure and if you can even remotely influence this quantity, you can learn how to, let's say, maximize it. And the quantity can measure success on the task. Or so this paradigm was used in various computer games, was used to beat humans in go. It was used in the game of Dota to get to the professional level. The paradigm has several shortcomings. So in a sense, as of today, it requires insane amount of data. So the rates are of tens of thousands of years of experience, virtual experience. So let's say in case of GO or in case of dota, agent is playing for so long for millennias, that's the requirement if you really want to get top raw superhuman performance. In case of robotics, it seems to be very hard to apply the same paradigm to the physical robots. I mean, it is possible you can try also to go through the path of decreasing number of interactions. So I would say that's one of the meaningful directions to just minimize the try to improve algorithms to minimize number of interactions. And one of the approaches was, let's say done at Google Stoop, let's say build form of arms to pick an object. But still there are few fundamental issues. So as long as once you are moving to the real world, it's actually not that easy to even get diversity of data. I mean, you cannot take your arm and go to a waterfall. Other part of the assumption is that you have to be able to reset entire thing from scratch. And also say once you are moving some objects, it can fall over and then you have to build some contraption mechanism to bring it back. So I started myself to think that maybe the paradigm of reinforcement learning, instead of being close to actually what is happening when the human is learning, is actually closer to what is happening during evolution. Like learning that you have a really gigantic population, it has huge number of the interactions, rather than a human kind of thinking through what truly happened and what should be the outcomes. So then it is also kind of natural to me that in case of evolution, like learning, you have, or let's say in case of human learning, there are essentially two stages. There is a stage that takes this gigantic, gigantic amount of data, which is evolution. It actually, it Is evolution is reinforcement learning. It's like going to survive or die. And this stage is powerful enough to create our brains that can then rapidly learn. So I was thinking, you know, it's actually not such a bad thing if we can learn in simulation to slightly add to almost majority of the task and then in reality there is quite rapid adaptation. So that's what we did. The interesting thing is when we so the task is we took robotics hand and we are reorienting objects and let's say speak in a second about the difficulty of this task. But solving the task itself takes something maybe like three years of virtual experience. But then to get to the capabilities that allow you for the transfer that takes another 97 years.

Oh wow. Yeah. So the generalization to create a model that creates this kind of or response to adaptations takes the longest amount of time.

Correct? Yeah.

Awesome. And just to kind of clarify a few things. So like when you're doing this virtual simulation, we've talked already on the podcast about where deep learning fits into the spectrum of AI techniques and we've talked about even like masks r CNN in robotics context. But we haven't really dove into reinforce learning yet. So I was wondering if you could just kind of give us a brief introduction to when you're doing the virtual learning and these kind of two stages. So you mentioned the first stage where it's kind of learning a task and then the second stage where you're attempting to make it more adaptive. What is the process that you're actually doing there?

So let me first describe what reinforcement learning is.

That'd be great.

So reinforcement learning, say framework of teaching an agent to maximize amount of reward. You can think about it a little bit like training a dog with a treat. So when the dog is doing good stuff, you are giving the dog treats and then he does more of the stuff that you want. And that's the way more or less how you train the computer to become best in all these games. So it's completely up to you when you define that you're going to give a treat.

Yeah. So there's a feedback that happens.

Correct. So agent itself effectively some network that consumes observations. You can think as of an analogy to a human. It's like input to eyes, ears, nose touch and so on. Network supposed to produce actions. So in our case it would be no electric signal. The nerves to decide how to move, let's say limbs or so. And the network itself attempts to maximize reward. The system has a chance only to be successful if from time to time it gets a reward. So that's a kind of, let's say, let me little bit downplay reinforcement learning. So the situations that wouldn't work is if you give a treat to a dog after it lands on Mars. Okay, it's okay. It will just never happen. It has to from time to time. It starts at first with very random strategy and then gradually it attempts to get more and more of the treats. So that's pretty much the paradigm of reinforcement learning.

And that's what you would be doing kind of in that first stage where you're teaching a specific. Is that right?

Correct. So let me briefly describe the task. So we haven't knew much about the robotics. Over a year ago we went to robotics conference. We asked people what are the things that are impossible or very hard to do in the classical robotics. And people are saying anytime when you have large number of degrees of freedom, it's very hard to control. When there is a lot of interactions, when you touch many things in many places simultaneously, that's also very hard to model. And there is when it is way easier when you are in the open space and you are not touching anything. Okay. Or if the problem somehow can be simplified to one or two dimensional problems. And there are some, let's say closed form solutions. But in case of robotics, Scanbotxcan has, let's say the one that we bought has 24 degrees of freedom. We also kind of knew that the task is solvable because human can solve it. So we wanted to have a hope for the success. The end task is you take an object. In our case, we demonstrated it on two different objects, which is one is a block, one is some octagonal prism and task is move it around to the new desired position.

Like in a robotic hand.

In the robotic hand. So as I said, we were able to train it actually a while ago already in the simulation to achieve it. But then during deployment it didn't work at all. Like literally not at all. Despite easily being able to solve it in the simulation. Typical response of let's say the typical approach is let's just get the simulation closer and closer and closer and closer to reality. And I would say that helps. But the problem is that with sufficiently complicated systems like the hand, and it's actually impossible to model everything. So since hand has tendons, tendons stretch, it has a rubber, rubber deforms. Also the shape that you have in the simulation actually doesn't even correspond exactly to the real shape. And when there is a lot of interactions, the difference in the given place, you Are touching versus not touching might cause the object to pivot, let's say slip over and so on. So that's why Sphinx do not want to transfer.

So it's like a lot of very small kind of differences in how things are touched or moved can create a whole different outcome.

Correct. In the sense that the fundamental idea that allows us for, let's say, adaptation to reality. This is extremely simple what we did. So in a sense, the initial approach is you have this single simulation. You can think about it like a single universe in which you're training and then you're asking, here is an alternative universe in which you want to actually verify the performance. And we are just instead saying if you will have entire distribution, many universes and network has a capability to encode its or let's say try to distinguish them, then it essentially might force the network to try to discover what are the underlying properties. So let's say if we don't know exactly what's the weight of that cube or I mean, more or less, maybe we know, but it might be off if we have a network that just has a capability to, let's say true interaction, pass the information. And these are like a very common networks, recurrent neural networks. Then as we vary these parameters and it tries in the simulation on all of these instances, maximize the score. It implicitly actually does. It has to try to, based on the initial few seconds, try to find out what are these values. Second, there are a combination of two things. On one side it tries to be robust to some components, and on one side it tries to adapt to various things. And in a sense this is in the core of the idea to actually achieve the transfer to the reality.

Yeah, that's really interesting. So how long have you been kind of working towards this type of adaptation and how has the process gone? Have you made other attempts to make this transfer to reality that haven't worked as well?

Yes. So entire project more or less took us to one year, let's say. I mean, earlier on there was, let's say maybe five people. Later on in the project there is maybe closer to, let's say 15 or so. So I would say average 10 human years. There were many, many attempts internally and many of the things that we tried, they partially worked and so on. But I mean, the way how the team is more or less organized is once we agree on the goal, try to, let's say, gauge people and ask what do you think is the best way to solve the problem? And in that sense, as I said, people to Some extent are on many axis they are smarter than me and they might sometimes better know than me was the best approach. When you're in the situation that there are multiple competitive approaches, they are also becoming closer to the truth what really works. So I mean it's often the case that idea, as long as it is let's say sound, it will show signs of life. But it doesn't mean that this the really ultimate solution. It might be sufficient for, you know, to present the conference or so. But our goal is truly to solve problem and actually get to the solutions that we think we can push forward.

So when I was looking through the OpenAI website and kind of looking into some of some of what you're trying to achieve, I kept coming across this statement about safe applications of AI. And I know we've talked in the past about AI ethics and other things. I was wondering if you could briefly talk about safe AI how you see that and maybe what counter examples to safe AI would be.

So there are several problems into it. One problem is question how to ensure that the system will be achieving the goal in the way that we intended it to achieve. So I mean I gathered many kind of philosophical examples and so on and I want to go through some of them. It sounds somewhat foreign, but simultaneously we are starting to see that actually it is not as trivial to tell actually system what to do because it is completely abstracted from our values, ethics and so on. So in essence you tell someone make money. It's like the best way to make money. Steal a car, sell drugs, all the stuff that is really that you wouldn't intend the system to do. It's actually the best things we probably.

Don'T want robots doing.

Yeah, I mean even you don't need robots for it. To give you some example, let's say if you are really insanely clever about stock market, you can, I'm saying insanely clever. And you know you have superhuman capabilities and what you truly care about is to maximize profit. You can cause a war in the country and short the stocks and it is completely valid strategy if that's the quantity to reach an objective. To reach the objective. And in some sense you can say that the systems that we are training, you can say it has little bit like a profile of psychopath. It only cares about the one thing and one thing only. That's literally how we optimize them. They are completely abstracted away from everything else and they want just these treats, treats, treats. And they actually don't know even about the things that are really important to us. The question is what's the way even to inject what we want? What is our ethics? And I mean I would say there are multiple axes into safety. So just told you about one, which is more or less called misspecification, means specify something, but it's actually something slightly different that you really wanted. And we can see it even in some computer games that he has the system to maximize score in the game. But truly you would like ask the system to finish the game and then it finds some bug on some level of the game and keeps on staying there, let's say generating a lot of points, but actually doesn't progress anymore in the game. I mean there are other axes, which is how to make the systems robust to adversaries and to give some concrete examples. So let's say there was, there was a Twitter bot release by Microsoft, Tanya and this folks from Microsoft. There is no doubt there's like a lot of very clever researchers and so on. So can say that it's like they thought through various scenarios. Okay. But despite, I'd say thinking it through, turns out that the bot within several hours was hijacked and repurposed since it started saying very offensive things on Twitter. So you might ask, is it the case that as the systems will become smarter, will they be less prone to it? I actually think that it might be due to the overall increased complexity, the surface area will just increase.

Yeah, it's almost. Even if you're trying to misspecify an objective, there's the space of objectives is larger, it's more complex.

So I would say from perspective of pursuing SAP AGI, there are also per se, three main things. So one, in order to achieve it, I mean you have to, let's say, work toward capabilities. Second one is you have to work towards safety. So what we just discussed. And the third one is a question. Let's say even if we would have it today, what's the step in terms of a policy? What should we do with it? And I would say also all these three components actually fit into each other. So capabilities indicate actually maybe what the shape of AGI will be. Therefore what are the ways to actually inject our ethics and so on the safety work and also it helps the policy people. So I'm saying like all these topics, they fit into each other.

Yeah, that makes sense. So I'd love to switch directions here for the last bit of the show and kind of get your perspective. So you mentioned a lot of really exciting things around robotics and a lot of that involved kind of large scale computing and lots of large simulations. I was wondering, for people that are out there trying to get their hands dirty with some of these techniques, maybe it's reinforcement learning or other things. What are some good ways for people to kind of get their hands dirty and start working on problems that are interesting, but maybe they aren't able to run these large scale simulations and that sort of thing?

Yeah, I mean, there are incredible materials online. So I would just go first through all Coursera Udacity lectures from Berkeley, Stanford, follow all the homeworks. So I would say that's step one. There's also great books. I mean, there is a book by Ian Deep learning book. There is reinforcement learning book by Richard Sutton. Yeah. And so I would say that would be my starting point. And I think it's actually quite important to get very strong fundamentals because in this sense, by default, when you run your models first they do not work at all. And then the question is, so what do you do? And the simpler models, the fewer tricks or steps you have to do and you have to familiarize yourself with them, the harder models, the larger number of these steps. So it's very likely that at first you don't know any of them. If you need to do 10 things, you are less likely to succeed versus if you need to do two things. I would really recommend to go through fundamentals instead of jumping right away to the most difficult architectures. And I would really recommend as much as you can to implement things from scratch.

Yeah, that's a great point. I know I've had some experiences in the past where some research comes out, it's really interesting. And you can go to a GitHub repo within a number of days and there's open source architecture there, but you try to run it and you see all these weird behavior and maybe it behaves differently than you would expect. But it's really hard if you don't know the fundamentals, to dive into the debug and advance. So, yeah, that's great. I know. I appreciated what you've said about any one person, even yourself, who has advanced a lot in this field, doesn't have all of the pieces of knowledge to perform successful research or to advance a project. I was wondering at OpenAI, how do you kind of structure your teams and what do you look for when you're kind of putting together a team so that you have a variety of experience and perspectives to actually give a good response result?

So different teams, they have a little bit different values and they are differently organized. I can speak about robotics. It's extremely important to have people who are good team players. I would say also when we hire people we hired based on being incredible in something, it doesn't need to be exactly what they will be working on or so it's like more or less you want to verify brilliancy and that's a sign that a person can adapt whatever is needed. I also, let's say so in a sense want people to be able from day zero to contribute. Still, let's say I encourage to spend one day per week on let's say, do arbitrary learning. We have, let's say internally curriculum with simple stuff. I like people who are excited about their resolving problems. So in the sense when it comes to difficult problems, as for instance, as the last project, it is very common that the time in the middle is when everything is extremely difficult. And you need that people who have this, let's say internal energy that they can still push it through and get it through.

Yeah. So persistence and motivation. A passion for the problem.

Correct. Yeah.

Awesome. Well, I'd love to take time if there's anything where can people find out more about OpenAI and are there any kind of open source projects or papers or efforts that you'd like to kind of share and we'll for sure post those in the show links and everything.

We are quite frequently releasing what we are able to build on our blog. So that's@the OpenAI.com website. We are also quite active on Twitter. So it's twitter.com OpenAI third word. You ask about the research proposals or so we posted, I think even twice. Let's say a bunch of ideas for projects for people if they want to pursue it, receive an indication of the level of difficulty. That might be a place to start.

Awesome. And where can we find that?

That's also on our website.

Awesome. Well, I really appreciate you taking time. I know that you must be busy and you're getting ready for your talk, so I'll let you get back to that. But thank you so much for joining us. It was a really great conversation.

Thank you, Daniel.

All right, thank you for tuning into this episode of Practically I. If you enjoyed this show, do us a favor, go on itunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do. Share the show with a friend if you enjoy it. And bandwidth for changelog is provided by Fastly. Learn more@fastly.com and we catch our errors before our users do here at Changelog because of Rollbar. Check them out@robot.com changelog and we're hosted on Linode Cloud Servers. Head to lino.com changelog check them out. Support this show this episode is hosted by Daniel Whitenack and Chris Benson. Editing is done by Tim Smith, the music is by Breakmaster Cylinder and you can find more shows just like this@change law.com when you go there, pop in your email address. Get our weekly email keeping you up to date with the news and podcast for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.