Sergey Levine: Robotics and Machine Learning | Lex Fridman Podcast #108

The following is a conversation with Sergey Levine, a professor at Berkeley and a world class researcher in deep learning, reinforcement learning, robotics and computer vision, including the development of algorithms for end to end training of neural network policies that combine perception and control, scalable algorithms for inverse reinforcement learning and in general, deep RL algorithms. Quick Summary of the ads 2 sponsors Cash App and ExpressVPN please consider supporting the podcast by downloading Cash App and using code lexpodcast and signing up at expressvpn.com lexpod Click the links Buy the Stuff it's the best way to support this podcast and in general, the journey I'm on. If you enjoy this thing, subscribe on YouTube, review it with five stars on Apple Podcast, follow on Spotify, support it on Patreon, or connect with me on Twitter lexfriedman. As usual, I'll do a few minutes of ads now and never any ads in the middle that can break the flow of the conversation. This show is presented by Cash App, the number one finance app in the App Store. When you get it, use code Lex Podcast Cash App lets you send money to friends, buy Bitcoin and invest in the stock market with as little as $1. Since Cash App does fractional share trading, let me mention that the order execution algorithm that works the scenes to create the abstraction of the fractional orders is an algorithmic marvel. So big props to the Cash App engineers for taking a step up to the next layer of abstraction over the stock market, making trading more accessible for new investors and diversification much easier. So again, if you get Cash App from the App Store, Google Play and use the code Lex podcast, you get $10 and Cash App will also donate $10 to FIRST, an organization that is helping to advance robotics and STE for young people around the world. This show is also sponsored by ExpressVPN. Get it@expressvpn.com Lexpod to support this podcast and to get an extra three months free on a one year package. I've been using ExpressVPN for many years. I love it. I think ExpressVPN is the best VPN out there. They told me to say it, but it happens to be true. In my humble opinion, it doesn't log your data. It's crazy fast and it's easy to use. Literally just one big power on button. Again, it's probably obvious to you, but I should say it again, that it's really important that they don't log your data. It works on Linux and every other operating system, but Linux of course is the best operating system Shout out to my favorite flavor, Ubuntu Mate 20.04. Once again, get it@expressvpn.com Lexpod to support this podcast and to get an extra three months free on a one year package. And now, here's my conversation with Sergey Levine. What's the difference between a state of the art human such as you and I? Well, I don't know if we qualify as state of the art humans, but a state of the art human and a state of the art robot?

That's a very interesting question. Robot capability is, I think it's a very tricky thing to understand because there are some things that are difficult that we wouldn't think are difficult, and some things that are easy that we wouldn't think are easy. And there's also a really big gap between capabilities of robots in terms of hardware and their physical capability and capabilities of robots in terms of what they can do autonomously. There is a little video that I think robotics researchers really like to show, especially robotics learning researchers like myself from 2004 from Stanford, which demonstrates a prototype robot called the PR1. And the PR1 was a robot that was designed as a home assistance robot. And there's this beautiful video showing the PR1 tidying up a living room, putting away toys, and at the end, bringing a beer to the person sitting on the couch, which looks really amazing. And then the punchline is that this robot is entirely controlled by a person. So in some ways, the gap between a state of the art human and a state of the art robot, if the robot has a human brain, is actually not that large. Now, obviously, human bodies are sophisticated and very robust and resilient in many ways. But on the whole, if we're willing to spend a bit of money and do a bit of engineering, we can kind of close the hardware gap almost. But the intelligence gap, that one is very wide.

And when you say hardware, you're referring to the physical sort of the actuators, the actual body of the robot, as opposed to the hardware on which the cognition, the hardware of the nervous system.

Yes, exactly. I'm referring to the body rather than the mind. So that means that the work is cut out for us while we can still make the body better. We kind of know that the big bottleneck right now is really the mind.

And how big is that gap? How big is the difference in your sense of ability to learn, ability to reason, ability to perceive the world between humans and our best robots?

The gap is very large. And the gap becomes larger the more unexpected events can happen. In the world. So essentially, the spectrum along which you can measure the size of that gap is the spectrum of how open the world is. If you control everything in the world very tightly, if you put the robot in, like, a factory and you tell it where everything is and you rigidly program its motion, then it can do things, you know, one might even say in a superhuman way. It can move faster, it's stronger, it can lift up a car and things like that. But as soon as anything starts to vary in the environment, now it'll trip up. And if many, many things vary like they would like in your kitchen, for example, the then things are pretty much like, wide open.

Now, again, we're going to stick a bit on the philosophical questions, but how much on the human side of the cognitive abilities, in your sense, is nature versus nurture? So how much of it is a product of evolution, and how much of it is something we learn from sort of scratch from the day we're born?

I'm going to read into your question as asking about the implications of this for AI. Of course, I'm not a biologist. I can't really speak authoritatively.

And to linger on it, if it's all about learning, then there's more hope for AI.

So the way that I look at this is that, well, first, of course, biology is very messy. And if you ask the question, how does a person do something? Or how does a person's mind do something? You can come up with a bunch of hypotheses, and oftentimes you can find support for many different, often conflicting hypotheses. One way that we can approach the question of what the implications of this for AI are is we can think about what's sufficient. So maybe a person is, from birth, very, very good at some things, like, for example, recognizing faces. There's a very strong evolutionary pressure to do that. If you can recognize your mother's face, then you're more likely to survive, and therefore people are. But we can also ask, what's the minimum sufficient thing? And one of the ways that we can study the minimum sufficient thing is we could, for example, see what people do in unusual situations if you present them with things that evolution couldn't have prepared them for. Our daily lives actually do this to us all the time. We didn't evolve to deal with automobiles and space flight and whatever. So there are all these situations that we can find ourselves in, and we do very well. There's like, I can give you a joystick to control a robotic arm, which you've never used before, and you might be pretty bad for the first couple of seconds, but if I tell you like your life depends on using this robotic arm to like open this door, you'll probably manage it. Even though you've never seen this device before, you've never used a joystick control list and you'll kind of muddle through it. And that's not your evolved natural ability, that's your flexibility or your adaptability. And that's exactly where current robotic systems really kind of fall flat.

But I wonder how much general almost what we think of as common sense pre trained model is underneath all of that. So that ability to adapt to a joystick requires you to have a kind of, you know, I'm human, so it's hard for me to introspect all the knowledge I have about the world. But it seems like there might be an iceberg underneath of the amount of knowledge we actually bring to the table. That's kind of the open question.

I think there's absolutely an iceberg of knowledge that we bring to the table. But I think it's very likely that that iceberg of knowledge is actually built up over our lifetimes because we have, you know, we have a lot of prior experience to draw on. And it kind of makes sense that the right way for us to, you know, to optimize our efficiency, our evolutionary fitness and so on is to utilize all that experience to build up the best iceberg we can get. And that's actually one. While that sounds an awful lot like what machine learning actually does, I think that for modern machine learning, it's actually a really big challenge to take this unstructured mass of experience and distill out something that looks like a common sense understanding of the world. And perhaps part of that is it's not because something about machine learning itself is broken or hard, but, but because we've been a little too rigid in subscribing to a very supervised, very rigid notion of learning. Kind of the input, output X is good, wise sort of model. And maybe what we really need to do is to view the world more as a mass of experience that is not necessarily providing any rigid supervision, but sort of providing many, many instances of things that could be. And then you take that and you distill it into some sort of common sense understanding.

I see. Well, you're painting an optimistic, beautiful picture, especially from the robotics perspective, because that means we just need to invest and build better learning algorithms, figure out how we can get access to more and more data for those learning algorithms to extract signal from and then accumulate that iceberg of knowledge. It's a beautiful picture, It's a hopeful one.

I think it's potentially a little bit more than just that. And this is where we perhaps reach the limits of our current understanding. But one thing that I think that the research community hasn't really resolved in a satisfactory way is how much it matters where that experience comes from. Do you just download everything on the Internet and cram it into essentially the 21st century analog of the giant language model and see what happens? Or does it actually matter whether your machine physically experiences the world in the sense that it actually attempts things, observes the outcome of its actions, and kind of augments its experience that way that.

It chooses which parts of the world it gets to interact with and observe and learn from?

Right. It may be that the world is so complex that simply obtaining a large mass of sort of IID samples of the world is a very difficult way to go. But if you are actually interacting with the world and essentially performing this sort of hard, negative mining by attempting what you think might work, observing the sometimes happy and sometimes sad outcomes of that, and augmenting your understanding using that experience, and you're just doing this continually for many years, maybe that sort of data in some sense is actually much more favorable to obtaining a common sense understanding. One reason we might think that this is true is that what we associate with common sense, or lack of common sense, is often characterized by the ability to reason about kind of counterfactual questions like, if I were to here, this bottle of water is sitting on the table, everything is fine. If I were to knock it over, which I'm not going to do, but if I were to do that, what would happen? And I know that nothing good would happen from that. But if I have a bad understanding of the world, I might think that that's a good way for me to gain more utility. If I actually go about my daily life doing the things that my current understanding of the world suggests will give me high utility. In some ways, I'll get exactly the right supervision to tell me not to do those bad things and to keep doing the good things.

So there's a spectrum between IID random walk through the space of data and what we humans do. Well, I don't even know if we do it optimal, but there might be beyond what? So this open question that you raised, where do you think systems, intelligent systems that would be able to deal with this world fall, can we do pretty well by reading all of Wikipedia, sort of randomly sampling it like Language models do or do we have to be exceptionally selective and intelligent about which aspects of the world we interact with?

So I think this is first an open scientific problem and I don't have a clear answer, but I can speculate a little bit. And what I would speculate is that you don't need to be super, super careful. I think it's less about being careful to avoid the useless stuff and more about making sure that you hit on the really important stuff. So perhaps it's okay if you spend part of your day just guided by your curiosity visiting interesting regions of your state space, but it's important for you to every once in a while make sure that you really try out the solutions that your current model of the world suggests might be effective and observe whether those solutions are working as you expect or not. And perhaps some of that is really essential to have kind of a perpetual improvement loop. This perpetual improvement loop is really like, that's really the key that's going to potentially distinguish the best current methods from the best methods of tomorrow. In a sense.

How important do you think is exploration or total out of the box thinking? Exploration in this space does you jump to totally different domains. So you kind of mentioned there's an optimization problem. You explore the specifics of a particular strategy, whatever the thing you're trying to solve. How important is it to explore totally outside of the strategies that have been working for you so far. What's your intuition there?

Yeah, I think it's a very problem dependent kind of question. And I think that that's actually, in some ways that question gets at one of the big differences between sort of the classic formulation of a reinforcement learning problem and some of the sort of more open ended reformulations of that problem that have been explored in recent years. So classically, reinforcement learning is framed as a problem of maximizing utility, like any kind of rational AI agent. And then anything you do is in service to maximizing that utility. But a very interesting kind of way to look at. I'm not necessarily saying this is the best way to look at it, but an interesting alternative way to look at these problems is as something where you first get to explore the world however you please, and then afterwards you will be tasked with doing something. And that might suggest a somewhat different solution. So if you don't know what you're going to be tasked with doing, and you just want to prepare yourself optimally for whatever your uncertain future holds, maybe then you will choose to attain some sort of coverage, build up sort of an arsenal of cognitive Tools, if you will, such that later on when someone tells you, now your job is to fetch the coffee for me, you'll be well prepared to undertake that task.

And you see that as the modern formulation of the reinforcement learning problem, as the kind of, the more multitask, the general intelligence kind of formulation.

I think that's one possible vision of where things might be headed. I don't think that's by any means the mainstream or standard way of doing things. And it's not like if I had.

To, but I like it. It's a beautiful vision. So maybe actually take a step back. What is the goal of robotics? What's the general problem robotics we're trying to solve? You actually kind of painted two pictures here, one of sort of the narrow one is the general. What in view is the big problem of robotics? Again, ridiculously philosophical, high level questions.

I think that, you know, maybe there are two ways I can answer this question. One is there's a very pragmatic problem which is like, what would make robots? What would sort of maximize the usefulness of robots? And there the answer might be something like a system where a system that can perform whatever task a human user sets for it within the physical constraints. Of course, if you tell it to teleport to another planet, it probably can't do that. But if you ask it to do something that's within its physical capability, then potentially with a little bit of additional training or a little bit of additional trial and error, it ought to be able to figure it out in much the same way as a human teleoperator ought to figure out how to drive the robot to do that. That's kind of the very pragmatic view of what it would take to kind of solve the robotics problem, if we will. But I think that there is a second answer, and the answer is a lot closer to why I want to work on robotics, which is that I think it's less about what it would take to do a really good job in the world of robotics, but more the other way around, what robotics can bring to the table to help us understand artificial intelligence.

So your dream fundamentally is to understand intelligence?

Yes, I think that's the dream for many people who actually work in this space. I think that there's something very pragmatic and very useful about studying robotics. But I do think that a lot of people that go into this field, actually the things that they draw inspiration from are the potential for robots to help us learn about intelligence and about ourselves.

So that's fascinating that robotics is Basically the space by which you can get closer to understanding the fundamentals of artificial intelligence. So what is it about robotics that's different from some of the other approaches? So if we look at some of the early breakthroughs in deep learning or in the computer vision space and then natural language processing, there's really nice clean benchmarks that a lot of people competed on and thereby came up with a lot of brilliant ideas. What's the fundamental difference to you between computer vision purely define an imagenet, and kind of the bigger robotics problem?

So there are a couple of things. One is that with robotics, you kind of have to take away many of the crutches. So you have to deal with both the particular problems of perception control and so on, but you also have to deal with the integration of those things. And classically, we've always thought of the integration as kind of a separate problem. So a classic kind of modular engineering approach is that we solve the individual subproblems, then wire them together, and then the whole thing works. And one of the things that we've been seeing over the last couple of decades is that, well, maybe studying the thing as a whole might lead to just like, very different solutions than if we were to study the parts and wire them together. So the integrative nature of robotics research helps us see different perspectives on the problem. Another part of the answer is that with robotics, it casts a certain paradox into very clever relief. So this is sometimes referred to as a Moravak's paradox, the idea that in artificial intelligence, things that are very hard for people can be very easy for machines, and vice versa, things that are very easy for people can be very hard for machines. So integral and differential calculus, pretty difficult to learn for people. But if you program a computer do it, it can derive derivatives and integrals for you all day long without any trouble. Whereas some things, like drinking from a cup of water, very easy for a person to do, very hard for a robot to deal with. And sometimes when we see such blatant discrepancies, that gives us a really strong hint that we're missing something important. So if we really try to zero in on those discrepancies, we might find that little bit that we're missing. And it's not that we need to make machines better or worse at math and better at drinking water, but just that by studying those discrepancies, we might find some new insight.

So that could be, that could be in any space. It doesn't have to be robotics. But you're saying, I mean, it's kind of interesting that robotics seems to have a lot of those discrepancies. So the, the, the hans more of a paradox is probably referring to the space of the physical interaction. Like you said, object manipulation, walking, all the kind of stuff we do in the physical world. How do you make sense if you were to try to disentangle the Maravox paradox, why is there such a gap in our intuition about it? Why do you think manipulating objects is so hard from everything you've learned from applying reinforcement learning in this space?

Yeah, I think that one reason is maybe that for many of the other problems that we've studied in AI and computer science and so on, the notion of input, output and supervision is much, much cleaner. So computer vision, for example, deals with very complex inputs, but it's comparatively a bit easier, at least up to some level of abstraction, to cast it as a very tightly supervised problem. It's comparatively much, much harder to cast robotic manipulation as a very tightly supervised problem. You can do it. It just doesn't seem to work all that well. So you could say that, well, maybe we get a labeled data set where we know exactly which motor commands to send and then we train on that. But for various reasons that's not actually such a great solution. And it also doesn't seem to be even remotely similar to how people and animals learn to do things. Because we're not told by our parents, here's how you fire your muscles in order to walk. We do get some guidance, but the really low level detailed stuff we figure out mostly on our own.

And that's what you mean by tightly coupled, that every single little sub action gets a supervised signal of whether it's a good one or not.

Right. So while in computer vision you could sort of imagine up to a level of abstraction that maybe somebody told you this is a car and this is a cat and this is a dog in motor control, it's very clear that that was not the case.

If we look at sort of the subspaces of robotics that again, as you said, robotics integrates all of them together and we get to see how this beautiful mess interplays. So there's nevertheless still perception. So it's the computer vision problem, broadly speaking, understanding the environment, then there's also, maybe you can correct me on this kind of categorization of the space. Then there's prediction in trying to anticipate what things are going to do into the future in order for you to be able to act in that world. And then there's also this game Theoretic aspect of how your actions will change the behavior of others in this kind of space. What. And this is bigger than reinforcement learning. This is just broadly looking at the problem of robotics. What's the hardest problem here? Or is there, or is what you said true, that when you start to look at all of them together, that's a whole nother thing. Like you can't even say which one individually is harder because all of them together, you should only be looking at them all together.

I think when you look at them all together, some things actually become easier, and I think that's actually pretty important. So we had, back in 2014, we had some work, basically our first work on end to end, reinforced learning for robotic manipulation skills from vision, which at the time was something that seemed a little inflammatory and controversial in the robotics world. But other than the inflammatory and controversial part of it, the point that we were actually trying to make in that work is that for the particular case of combining perception and control, you could actually do better if you treat them together than if you try to separate them. And the way that we tried to demonstrate this is we picked a fairly simple motor control task where a robot had to insert a little red trapezoid into a trapezoidal hole. And we had our separated solution, which involved first detecting the hole using a pose detector and then actuating the arm to put it in, and then our intent solution, which just mapped pixels to torques. And one of the things we observed is that if you use the M10 solution, essentially the pressure on the perception part of the model is actually lower. It doesn't have to figure out exactly where the thing is in 3D space. It just needs to figure out where it is distributing the errors in such a way that the horizontal difference matters more than the vertical difference, because vertically it just pushes it down all the way until it can't go any further. And there perceptual errors are a lot less harmful, whereas perpendicular to the direction of motion, perceptual errors are much more harmful. So the point is that if you combine these two things, you can trade off errors between the components optimally to best accomplish the task, and the components can actually be weaker while still leading to better overall performance.

That's a profound idea. In the space of pegs and things like that, it's quite simple. It almost is tempting to overlook. But that seems to be, at least intuitively, an idea that should generalized to basically all aspects of perception and control. Of course, that one strengthens the other.

Yeah. And people who have studied Sort of perceptual heuristics in humans and animals find things like that all the time. So one very well known example is something called the gaze heuristic, which is a little trick that you can use to intercept a flying object. So if you want to catch a ball, for instance, you could try to localize it in 3D space, estimate its velocity, estimate the effect of wind resistance, solve a complex system of differential equations in your head, or you can maintain a running speed so the object stays in the same position as in your field of view. So if it dips a little bit, you speed up. If it rises a little bit, you slow down. And if you follow the simple rule, you'll actually arrive at exactly the place where the object lands and you'll catch it. And humans use it when they play baseball. Human pilots use it when they fly airplanes to figure out if they're about to collide with somebody. Frogs use this to catch insects, and so on and so on. So this is something that actually happens in nature. And I'm sure this is just one instance of it that we were able to identify just because all that scientists were able to identify because it's so prevalent. But there are probably many others.

Just so we can zoom in. As we talk about robotics, do you have a canonical problem, sort of a simple, clean, beautiful, representative problem in robotics that you think about when you're thinking about some of these problems? We talked about robotic manipulation. To me, that seems intuitively at least, the robotics community has converged towards that as a space that's the canonical problem. If you agree, then maybe do you zoom in in some particular aspect of that problem that you just like? Like, if we solve that problem perfectly, it'll unlock a major step in towards human level intelligence?

I don't think I have, like a really great answer to that. And I think partly the reason I don't have a great answer kind of has to do with the. It has to do with the fact that the difficulty is really in the flexibility and adaptability rather than in doing a particular thing really, really well. So it's hard to just say, like, oh, if you can, I don't know, shuffle a deck of cards as fast as a Vegas casino dealer, then you'll be very proficient. It's really the ability to quickly figure out how to do some arbitrary new thing well enough to move on to the next arbitrary thing.

But the source of newness and uncertainty. Have you found problems in which it's easy to generate new newnesses, new types.

Of newness yeah, So a few years ago, so if you'd asked me this question around 2016, maybe I would have probably said that robotic grasping is a really great example of that because it's a task with great real world utility. Like, you will get a lot of money if you can do it well.

What is robotic grasping? Picking up any object with a robotic hand.

Exactly. So you will get a lot of money if you do it well, because lots of people want to run warehouses with robots and it's highly non trivial because very different objects will require very different grasping strategies. But actually since then, people have gotten really good at building systems to solve this problem to the point where I'm not actually sure how much more progress we can make with that as the main guiding thing. But it's kind of interesting to see the kind of methods that have actually worked well in that space. Because robotic grasping classically used to be regarded very much as almost like a geometry problem. So people who have studied the history of computer vision will find this very familiar. It turns out that what works really well for robotic grasping, instantiated in many different recent works, including our own, but also ones from many other labs, is to use learning methods with some combination of either exhaustive simulation or actual real world trial and error. And it turns out that those things actually work really well. And then you don't have to worry about solving geometry problems or physics problems.

So what are just, by the way, on the grasping, what are the difficulties that have been worked on? So one is like the materials of things, maybe occlusions on the perception side, why is it such a difficult. Why is picking stuff up such a difficult problem?

Yeah, it's a difficult problem because the number of things that you might have to deal with or the variety of things that you have to deal with is extremely large. And oftentimes things that work for one class of objects won't work for other classes of objects. So if you get really good at picking up boxes and now you have to pick up plastic bags, you just need to employ a very different strategy. And there are many properties of objects that are more than just their geometry. It has to do with the bits that, that are easier to pick up, the bits that are harder to pick up, the bits that are more flexible, the bits that will cause the thing to pivot and bend and drop out of your hand, versus the bits that result in a nice secure grasp. Things that are flexible, things that if you pick them up the wrong way, they'll fall upside down and the contents will spill out. So there's all these little details that come up, but the task still kind of can be characterized as one task. There's a very clear notion of you did it or you didn't do it.

So in terms of spilling things, there creeps in this notion that starts to sound and feel like common sense reasoning. Do you think solving the general problem of robotics requires common sense reasoning, Requires general intelligence, this kind of human level capability of, like you said, be robust and deal with uncertainty, but also be able to sort of reason and assimilate different pieces of knowledge that you have? Yeah. What are your thoughts on the needs of common sense reasoning in the space of the general robotics problem?

So I'm going to slightly dodge that question and say that I think maybe actually it's the other way around is that studying robotics can help us understand how to put common sense into our AI systems. One way to think about common sense is that, and why our current systems might lack common sense is that common sense is an emergent property of actually having to interact with a particular world, a particular universe, and get things done in that universe. So you might think that, for instance, like an image captioning system, maybe it looks at pictures of the world and it types out English sentences, so it kind of deals with our world. And then you can easily construct situations where image captioning systems do things that defy common sense, like give it a picture of a person wearing a fur coat and we'll say it's a teddy bear. But I think what's really happening in those settings is that the system doesn't actually live in our world. It lives in its own world that consists of pixels and English sentences and doesn't actually consist of having to put on a fur coat in the winter so you don't get cold. So perhaps the reason for the disconnect is that the systems that we have now simply inhabit a different universe. And if we build AI systems that are forced to deal with all of the messiness and complexity of our universe, maybe they will have to acquire common sense to essentially maximize their utility. Whereas the systems we're building now don't have to do that. They can take some shortcuts.

That's fascinating. You've a couple times already sort of reframed the role of robotics in this whole thing. And for some reason, I don't know if my way of thinking is common, but I thought, like, we need to understand and solve intelligence in order to solve robotics. And you're kind of framing it as no. Robotics is one of the Best ways to just study artificial intelligence and build sort of like robotics is like the right space in which you get to explore some of the fundamental learning mechanisms. Fundamental sort of multimodal, multitask aggregation of knowledge mechanisms that are required for general intelligence. That's really interesting way to think about it. But let me ask about learning. Can the general sort of robotics, the epitome of the robotics problem, be solved purely through learning? Perhaps end to end learning, sort of learning from scratch as opposed to injecting human expertise and rules and heuristics and so on?

I think that in terms of the spirit of the question, I would say yes. I mean, I think that though in some ways it's maybe like an overly sharp dichotomy. I think that in some ways when we build algorithms, we at some point a person does something, yeah, hyperpress. There's always a person turned on the computer, a person implemented TensorFlow. But yeah, I think that in terms of the point that you're getting at, I do think the answer is yes. I think that we can solve many problems that have previously required meticulous manual engineering through automated optimization techniques. And actually, one thing I will say on this topic is I don't think this is actually a very radical or very new idea. I think people have been thinking about automated optimization techniques as a way to do control for a very, very long time. And in some ways what's changed is really more the name. So today we would say that, oh, my robot does machine learning, it does reinforcement learning. Maybe in the 1960s you'd say, oh, my robot is doing optimal control. And maybe the difference between typing out a system of differential equations and doing feedback linearization versus training a neural net, maybe it's not such a large difference, it's just pushing the optimization deeper and deeper into the thing.

Well, it's interesting you think that way, but especially with deep learning, that the accumulation of sort of experiences in data form to form deep representations starts to feel like knowledge as opposed to optimal control. This feels like there's an accumulation of knowledge through the learning process.

Yes. So I think that is a good point. That one big difference between learning based systems and classic optimal control systems is that learning based systems in principle should get better and better the more they do something. And I do think that that's actually a very, very powerful difference.

So if we look back at the world of expert systems and symbolic AI and so on, of using logic to accumulate expertise, human expertise, human encoded expertise, do you think that will have a role at some point, the you know, deep learning, machine learning, reinforcement learning has been. Has shown incredible results and breakthroughs and just inspired thousands, maybe millions of researchers. But, you know, there's this less popular now, but it used to be popular idea of symbolic AI. Do you think that will have a role?

I think in some ways the descendants of symbolic AI actually already have a role. So this is the highly biased history. From my perspective. You say that, well, initially we thought that rational decision making involves logical manipulation. So you have some model of the world expressed in terms of logic. You have some query like, what action do I take in order for X to be true? And then you manipulate your logical symbolic representation to get an answer. What that turned into somewhere in the 1990s is, well, instead of building kind of predicates and statements that have true or false values, we'll build probabilistic systems where things have probabilities associated and probabilities of being true and false. And that turned into Bayes nets. And that provided sort of a boost to what were really still essentially logical inference systems, just probabilistic logical inference systems. And then people said, well, let's actually learn the individual probabilities inside these models. And then people said, well, let's not even specify the nodes in the models. Let's just put a big neural net in there. But in many ways, I see these as actually kind of descendants from the same idea. It's essentially instantiating rational decision making by means of some inference process and learning by means of an optimization process. So in a sense, I would say, yes, that it has a place. And in many ways that place is. It already holds that place.

It's already in there. Yeah, it's just by different. It looks slightly different than it was before.

Yeah. But there are some things that we can think about that make this a little bit more obvious. Like if I train a big neural net model to predict what will happen in response to my robot's actions, and then I run probabilistic inference, meaning I invert that model to figure out the actions that lead to some plausible outcome. To me, that seems like a kind of logic. You have a model of the world, it just happens to be expressed by a neural net, and you are doing some inference procedure, some sort of manipulation on that model to figure out the answer to a query that you have.

It's the interpretability, it's the explainability, though, that seems to be lacking more so because the nice thing about sort of expert systems is you can follow the reasoning of the system that to us mere humans is Somehow compelling. It's just I don't know what to make of this fact that there's a human desire for intelligence systems to be able to convey in a poetic way to us why it made the decisions it did, like tell a convincing story. And perhaps that's like a silly human thing. Like we shouldn't expect that of intelligent systems. Like we should be super happy that there's intelligent systems out there. But if I were to sort of psychoanalyze the researchers at the time, I would say expert systems connected to that part, that desire of AI researchers for systems to be explainable. I mean, maybe on that topic, do you have a hope that sort of inference systems, of learning based systems will be as explainable as the dream was with expert systems, for example?

I think it's a very complicated question because I think that in some ways the question of explainability is kind of very closely tied to the question of performance. Why do you want your system to explain itself? Well, it's so that when it screws up, you can kind of figure out why it did it right. But in some ways that's a much bigger problem, actually. Like your system might screw up and then it might screw up in how it explains itself, or you might have some bug somewhere so that it's not actually doing what it was supposed to do. So maybe a good way to view that problem is really as a bigger problem of verification and validation, of which explainability is sort of one component.

I see. I just see it differently. I see explainability. You put it beautifully. I think you actually summarize the field of explainability. But to me, there's another aspect of explainability which is like storytelling that has nothing to do with errors or with. Uses errors as. As elements of its story, as opposed to a fundamental need to be explainable when errors occur. It's just that for other intelligence systems to be. In our world, we seem to want to tell each other stories. And that's true in the political world, that's true in the academic world, and that neural networks are less capable of doing that, or perhaps they're equally capable of storytelling is storytelling. Maybe it doesn't matter what the fundamentals of the system are. You just need to be a good storyteller.

Maybe one specific story I can tell you about in that space is actually about some work that was done by my former collaborator who's now a professor at MIT named Jacob Andreas. Jacob actually works in natural language processing, but he had this idea to do a little bit of work in reinforcement learning on how natural language can basically structure the internals of policies. Trained with rl and one of the things he did is he set up a model that attempts to perform some task that's defined by a reward function. But the model reads in a natural language instruction. So this is a pretty common thing to do in instruction following. So you tell it like, you know, go to the red house and then it's supposed to go to the red house. But then one of the things that Jacob did is he treated that sentence not as a command from a person, but as a representation of the internal kind of state of the mind of this policy, essentially. So that when it was faced with a new task, what it would do is it would basically try to think of possible language descriptions, attempt to do them and see if they led to the right outcome. So it would kind of think out loud like, I'm faced with this new task, what am I going to do? Let me go to the red house. Oh, that didn't work. Let me go to the blue room or something, let me go to the green plant. And once it got some reward it would say, oh, go to the green plant. That's what's working. I'm going to go to the green plant. And then you could look at the string that it came up with and that was a description of how it thought it should solve the problem. So you could basically incorporate language as internal state and you can start getting some handle on these kinds of things.

And then what I was kind of trying to get to is that also if you add to the reward function the convincingness of that story. So I have another reward signal of like people who review that story, how much they like it. So that initially that could be a hyperparameter, sort of hard coded, heuristic type of thing. But it's an interesting notion of the convincingness of the story becoming part of the reward function, the objective function of the explainability. In the world of sort of Twitter and fake news, that might be a scary notion that the nature of truth may not be as important as the convincingness of how convincing you are in telling the story around the facts. Well, let me ask the basic question. You're one of the world class researchers in reinforcement learning, deep reinforcement learning, certainly in the robotics space. What is reinforcement learning?

I think that what reinforcement learning refers to today is really just the kind of the modern incarnation of learning based control. So classically reinforcement learning has a much more narrow definition, which is that it's literally learning from reinforcement. Like the thing does something and then it gets a reward or punishment. But really, I think the way the term is used today is it's used to refer more broadly to learning based control. So some kind of system that's supposed to be controlling something and it uses data to get better.

And what does control mean? So this action is the fundamental element.

It means making rational decisions. And rational decisions are decisions that maximize.

A measure of utility and sequentially. So you made decisions time and time and time again. Now, like, it's easy to see that kind of idea in the space of maybe games, in the space of robotics. Do you see it bigger than that? Is it applicable? Like, where are the limits of the applicability of reinforcement learning?

Yeah, so rational decision making is essentially the encapsulation of the AI problem viewed through a particular lens. So any problem that we would want a machine to do, an intelligent machine, can likely be represented as a decision making problem. Classifying images is a decision making problem, although not a sequential one. Typically, you know, controlling a chemical plant is a decision making problem. Deciding what videos to recommend on YouTube is a decision making problem. And one of the really appealing things about reinforcement learning is if it does encapsulate the range of all these decision making problems. Perhaps working on reinforcement learning is one of the ways to reach a very broad swath of AI problems.

But what do you use the fundamental difference between reinforcement learning and maybe supervised machine learning?

So reinforcement learning can be viewed as a generalization of supervised machine learning. You can certainly cast supervised learning as a reinforced learning problem. You can just.