MIT Sloan: Intro to Machine Learning (in 360/VR)

The video you're watching now is in 360 Resolution, is not great, but we wanted to try something different. So if you're on a desktop or laptop, you can pan around with your mouse, or if you're on a phone.

Or tablet, you should be able to.

Just move your device to look around. Of course, it's best viewed with a VR headset. The video that follows is a guest lecture on machine learning that I gave at MIT Sloan course on the business of Artificial Intelligence. The lecture is non technical and intended to build intuition about these ideas amongst the business students in the audience. The room was a half circle, so we thought why not film the lecture in 360. We recorded a screencast of the slides and pasted it into the video so that the slides are more crisp. Let me know what you think and remember, it's an experiment. So this course is talking about the broad context, the impact of artificial intelligence, the global. There's global, which is the global impact of artificial intelligence. There's the business, which is when you have to take these fun research ideas that I'll talk about today. A lot of them are cool on toy examples. When you bring them to reality, you face real challenges, which is what I would like to really highlight today. That's the business part, when you want to make real impact, when you make these technologies a reality. So I'll talk about how amazing the technology is for a nerd like me, but also talk about how when you take that into the real world, what are the challenges you face? So machine learning, which is the technology at the core of artificial intelligence. We'll talk about the promise, the excitement that I feel about it, the limitations. We'll bring it down a little bit. What are the real capabilities of technology.

Where for the first time.

Really as a civilization exploring the meaning of intelligence?

It is if you pause for a.

Second and just think, you know, maybe many of you want to make money out of this technology, many of you want to save lives, help people, but also on the philosophical level, we get to explore what makes us human. So while I'll talk about the low level technologies, we also think about the incredible opportunity here. We get to almost psychoanalyze ourselves by trying to build versions of ourselves in the machine. All right, so here's the open question. How powerful is artificial intelligence? How powerful is machine learning that lies at the core of artificial intelligence? Is it simply a helpful tool, a special purpose tool to help you solve simple problems if you're. Which is what it currently is Currently machine learning, artificial Intelligence is a way if you can formally define the problem, you can formally define the tools you're working with, you can formally define the utility function, what you want to achieve with those tools. As long as you can define those things, we can come up with algorithms that can solve them. As long as you have the right kind of data, which is what I'll talk about. Data is key. And the question is into the future. Can we break past this very narrow definition of what machine learning can give us, which is solve specific problems, to something bigger, to where we approach the general intelligence that we exhibit as human beings when we're born, we know nothing and we learn quickly from very little data. The right answer is we don't know. We don't know. What are the limitations of technology? What kind of machine learning are there?

There are several flavors. The first two is what's really.

The first is what's achieved success today. Supervised learning. What I'm showing here on the left of the slide is the teachers is the data that is fed to the system and on the right is the students, which is the system itself for machine learning. So there's supervised learning. When everybody talks about machine learning today, for the most part they're referring to supervised learning, which means every single piece of data that is used to train the model is seen by human eyes. And those human eyes with an accompanying brain label that data in a way that makes it useful to the machine.

This is, this is critical because that's.

One the blue box. The human is really costly. So whenever every single piece of data that needs to be, that's used to train the machine needs to be seen by a human, you need to pay for that human. And second, you're limited to just the time. There's the amount of data necessary to label what it means to exist in this world is humongous. Augmented supervised learning is when you get machine to really to help you a little bit.

There's a few tricks there, but still.

It'S still only tricks. It's still the human is at the core of it. And the promise of future research that we're pursuing, that I'm pursuing and perhaps in the applications, if we get to discuss, or some of the speakers here get to discuss, they're pursuing in semi supervised and reinforcement learning where the human starts to play a smaller and smaller role in how much they get to annotate. They have to annotate the data. And the dream of the sort of.

Wizards of the dark arts of deep.

Learning are all excited about unsupervised learning that has very few actual successes in application in the real world today.

But it is the idea that you.

Can build a machine that doesn't require a human teacher, a human being to teach you anything fills us artificial intelligence researchers with excitement. There's a theme here. Machine learning is really simple. The learning system. In the middle, there's a training stage where you teach it something. All you need is some data, input data, and you need to teach it.

The correct output for that input data.

So you have to have a lot of pairs of input data and correct output. There'll be a theme of cats throughout this presentation. So if you want to teach a.

System.

Difference between a cat and a dog, you need a lot of images of cats. And you need to tell it that this is a cat. This bounding box here in the image is a cat. You have to give it a lot of images of dogs and tell it, okay, well, this, in this, in these pictures, they're dogs. And then there's a spelling mistake. On the second stage is the testing stage, when you actually give it new input data it's never seen before. And you hope that it has given for cat versus dog enough data to guess, is this new image that I've never seen before a cat or a dog? Now, one of the open questions you want to keep in mind is what in this world can we not model in this way?

What activity, what task, what goal.

I offer to you that there's nothing you can't model in this way. So let's think about what in terms of machine learning can be. So let's start small. What can be modeled in this way? First, on the bottom of the slide left is one to one mapping where the input is an image of a cat and the output is, is a label that says cat or dog. You can also do one to many where the image, the input is a image of a cat and the output is a story about that cat. A captioning of the image. You can, first of all, you can do the other way, many to one mapping where you give it a story about a cat and it generates an image. There's many to many. This is Google Translate. We translate a sentence from one language to another, and there's various flavors of that. Again, same theme here. Input data provided with correct output and then let it go into the wild where it runs on input data it hasn't seen before to provide guesses. And it's as simple as this. Whatever you can convert into one of the following four things. Numbers, vector of numbers. So a bunch of numbers, a sequence of numbers where the temporal dynamics matters. So like audio video where the sequence, the ordering matters or a sequence of vector numbers. Just a bunch of numbers. If you can convert it into numbers and I propose to you that there's nothing you can't convert into numbers if you can convert it to numbers, you can have a system learn to do it. And the same thing with the output, generate numbers, Vectors of numbers, Sequence of numbers or sequence of vectors and numbers.

First.

Is there any questions at this point? Well, we have a lot of fun slides to get through, but I'll, I'll pause every once in a while to make sure we're on the same page here. So what kind of input are we talking about? Just to fly through it images, so faces or medical applications for looking, looking at scans of different parts of the body to determine if the to diagnose any kind of medical conditions. Text so conversations, your texts, article, blog post for sentiment analysis. Question answering. So you ask it a question where the output you hope answers sound. So voice recognition, any kind of anything you could tell from audio time series data. So financial data, stock market, you can use it to predict anything you want about the stock market, including whether to buy or sell if you're curious. Doesn't work quite well as a machine learning application. Physical world. So cars or any kind of object.

Any kind of robot that exists in this world.

So location of where I am, location of where other things are, the actions of others that could be all input, all of it can be converted to numbers and the correct output, same thing, Classification, a bunch of numbers. Classification is saying is this a cat or a dog? Regression is saying to what degree I turn the steering wheel Sequence generating audio, generating video, generating stories, captioning text. Images generate anything you could think of as numbers. And at the core of it is a bunch of data. Agnostic machine learning algorithms. There's traditional ones, nearest neighbors, naive bay support, machine support, vector machines. A lot of them are.

Limited.

And I'll describe how. And then there's neural networks. There's nothing special and new about neural networks. And I'll describe exactly the very subtle thing that is powerful, that's always been there all along. And certain things have now been able to unlock that power about neural networks.

But it's still just the flavor of.

A machine learning algorithm. And the inspiration for neural networks, as Jonathan showed last time, is our human brain. It's perhaps why the media, perhaps why.

The hype is captivated by the idea.

Of neural networks is because you immediately Jump to this feeling like because there's this mysterious structure to them that scientists don't understand. Artificial neural networks, I'm referring to, and the biological ones, we don't understand them and the similarity captivates our minds. And we think, well, this approach is perhaps as limited as our, as limitless as our own human mind. But the comparison ends there. In fact, the artificial neuron, their artificial neural networks are much simpler computational units. At the core of everything is this neuron. This is a computational unit that does two very simple operations. On the left side, it takes a set of numbers as inputs. It applies weights to those inputs, sums them together, applies a little bias, and provides an output somewhere between 0 and 1. So you could think of it as a computational entity that gets excited when it sees certain inputs and gets totally turned off when it gets other kinds of inputs. So maybe this neuron with a 0.7, 0.6, 1.4 weights, it gets really excited when it sees pictures of cats and totally doesn't care about dogs. Some of us are like that. So that's the job of this neuron.

Is to detect cats. Now what?

The way you build an artificial neural network, the way you release the power that I'll talk about in the following slides about the applications, what can be achieved is just stacking a bunch of these together. Think about it. This is an extremely simple computational unit. So you need to sort of pause whenever we talk about the following slides and think that.

There'S a few slides.

That I'll show that say neural networks are amazing. I want you to think back to this slide that everything is built on top of these really simple addition operations with a simple nonlinear function applied at the end. Just a tiny math operation. We stack them together in a feed forward way so there's a bunch of layers. And when people talk about deep neural networks, it means there's a bunch of those layers. And then there's recurrent neural networks that are also a special flavor that's able to have memory. So as opposed to just pushing input into output directly.

And it's also able to do stuff.

On the inside in a loop where it remembers things. This is useful for natural language processing, for audio processing. Whenever the sequence is not, the length of the sequence is not defined. Ok, slide number one in terms of neural networks are amazing. This is perhaps for the math nerds, but also I want you to use your imagination. There's a universality to neural networks means that this simple computational unit on the.

Left is an input, on the right.

Is the output of this network with just a single hidden layer. It's called a hidden layer because it sits there in the middle of the input and the output layers. A single hidden layer with some number of nodes can represent any function, any function that means anything you want to build in this world. Everyone in this room can be represented with a neural network with a single hidden layer. So the power, and this is just one hidden layer. The power of these things is limitless. The problem, of course, is how do you find the network. So how do you build a network that is as clever as many of the people in this room? But the fact that you can build such a network is incredible, is amazing. I want you to think about that and the way you train a network.

So.

It'S born as a blank slate, some random weights assigned to the edges. Again, a network is represented. The numbers at the core, the parameters at the core of this network are the numbers on each of those arrows, each of those edges, and you start knowing nothing. This is a baby network. And the way you teach it something, unfortunately, currently, as I said, in a supervised learning mechanism, you have to give it pairs of input and output.

You have to give it pictures of.

Cats and labels on those pictures saying that they're cats. And the basic fundamental operation of learning is when you compute the measure of an error and you back propagate it to the network. What I mean, everything is easier with cats.

I apologize, I apologize.

Too many cats. And so the input here is a cat. And the neural network we trained, it's just guessing, it doesn't know. Say I don't know, it's guessing cat. Well, it happens to be right. So we have to. This is the measure of error.

Yes, you got it right.

And you have to back propagate that error. You have to reward the network for.

Doing a good job.

And all you do. What I mean by reward, there's weights on each of those edges. And so the node, the individual neurons that were responsible that back to that cat neuron, that cat neuron needs to.

Be rewarded for seeing the cat.

So you just increase the weights on the neurons that were associated with producing the correct answer.

Now you give it a picture of.

A dog and the neural network says cat. Well, that's an incorrect answer. So no, there's a high error needs to be back propagated through the network. So the weights that are responsible with classifying this picture as a cat need to be punished. They need to be decreased. Simple. And you just repeat this process over and over. This is what we do as Kids, when we're first learning, for the most part, we have two. We're also supervised learning machines in the sense that we have our parents and we have the environment, the world that teach us about what's correct and what's incorrect. And we back propagate this error and reward through our brain to learn. The problem is, as human beings, we don't need too many examples. And I'll talk about some of the.

Drawbacks of these approaches.

We don't need too many examples. You fall off your bike once or twice and you learn how to ride the bike. Unfortunately, neural networks need tens of thousands of times when they fall off the bike in order to learn how to not do it. That's one of the limitations. And one key thing I didn't mention here is when we refer to the.

Input data.

It'S when we refer to input data, we usually refer to sensory data, raw data. We have to represent that data in some clever way, in some deeply clever way where we can reason about it, whether it's in our brains or in the neural network. And a very simple example here to illustrate why representation of data matters. So the way you represent the data can make the discrimination of one class from another, a cat versus dog, either incredibly difficult or incredibly simple. Here is a visualization of the same kind of data in cartesian coordinates and polar coordinates on the right. You can just draw a simple line to separate the two. What you want is a system that's able to learn the polar coordinate representation versus the Cartesian representation automatically. And this is where deep learning has stepped in and revealed the incredible power.

Of this approach, which deep learning is.

The smallest circle there. It's a type of representational learning. Machine learning is the bigger circle second to the biggest. So this class is about the biggest circle AI includes robotics, includes all the fun things that are built on learning. And I'll discuss while machine learning, I think, will close this entire circle into one. But for now, AI is the biggest circle, then a subset of that is machine learning, and a smaller subset of that is representation learning. So deep learning is not only able to say, given a few examples of cats and dogs, to discriminate between a cat and a dog. It's able to represent what it means to be a cat. So it's able to automatically determine what are the fundamental units at the low level and the high level.

Talking about this is very play doh.

What it means to represent a cat, from the whiskers to the high level shape of the head, to the fuzziness and the Deformable aspects of the cat. Not a cat expert, but I hear these are the features of a cat.

That are essential to discriminate between a.

Cat and a dog. Learning those features as opposed to having to have experts. This is the drawback of systems that Jonathan talked about from the 80s and 90s where you had to bring in experts for any specific domain that you try to solve. You had to have them encode that information. Deep learning, this is simply the only big difference between deep learning and other methods is that it learns the representation for you. It learns what it means to be a cat. Nobody has to step in and help it figure out that cats have whiskers and dogs don't. What does this mean? The fact that it can learn these features, these whisker features, is as opposed to having 5 or 10 or 100 or 500 features that are encoded by brilliant engineers with PhDs. It can find hundreds of thousands, millions of features automatically. Hundreds of millions of features. So stuff that is, that can't be put into words or described. In fact, it's one of the limitations in neural networks is they find so many fundamental things about what it means to be a cat that you can't visualize what it really knows. It just seems to know stuff, and it finds that stuff automatically. What, what does this mean?

It's the.

The critical thing here is because it's able to automatically learn those hundreds of millions of features, it's able to utilize data. It doesn't start. The diminishing returns don't hit until, well, we don't know when they hit. The point is, with the classical machine learning algorithms, you start hitting a wall when you have tens of thousands of images of cats. With deep learning, you get better and better with more data. Neural networks are amazing.

Slide 2.

Here's a game, a simple arcade game where there's two paddles. They're bouncing a ball back and forth.

Okay, great.

You can figure out an artificial intelligence agent that can play this game. It can not even that well, just kind of. It kind of learns to do all right and eventually win. Here's the fascinating thing with deep learning, as opposed to encoding, the position of the paddles, the position of the ball, having an expert in this game, there's many come in and encode the physics of this game. The input to the neural network is the raw pixels of the game. So it's learning in the following way. You give it an evolution of the game, you give it a bunch of pixels. Pixels are images that built up of pixels. They're just numbers from 0 to 256. So there's this array of numbers that represent each image and then you give it several tens of thousands of images that represent a game.

So you have this stack of pixels.

And stack of images that represent a game. And the only thing you know, this giant stack of numbers, the only thing you know is at the end you won or lost.

That's it.

So based on that, you have to.

Figure out how to play the game.

You know nothing about games. You know nothing about colors or balls or paddles or winning or anything.

That's it. So this is.

Why is this amazing that it even works? And it works. It wins. It's amazing because that's exactly what we do as human beings. This is general intelligence. So I need you to pause and think about this. We'll talk about special intelligence and the usefulness and okay, there's cool tricks here.

And there that we can do to.

Get you an edge on your high frequency trading system. But this is general intelligence. General intelligence is the same intelligence we use as babies when we're born. What we get is an input, sensory input of image. Sensory input. Right now, all of us, most of us are seeing, hearing, feeling with touch. And that's the only input we get. We know nothing. And with that input we have to learn something. Nobody is pre teaching us stuff. And this is an example of that. A trivial example, but one of the first examples where this is truly working. Sorry to linger on this, but it's a fundamental fact. The fact that we have systems that now outperform human beings in these simple arcade games is incredible. This is the research side of things. But let me step back these again, the takeaways. That previous slide is why I think.

Machine learning is limitless.

In the future. Currently it's limited. Again, the representation of the data matters. And if you want to have impact, we currently can only tackle the small problems. What are those problems? Image recognition we can classify given the entire image of a leopard, of a boat, of a mite, with pretty good.

Accuracy.

Of what's in that image. That's image classification. What else? We can find exactly where in that image each individual object is. That's called image segmentation. Again, the process is the same. The learning system in the middle, a neural network. As long as you give it a set of numbers as input and the correct set of labels as output. It learns to do that for data it hasn't seen in the past. Let me pause a second and maybe if you have any questions. Does anyone have any Questions about the techniques of neural networks.

Yes. How do you.

Represent data, as you mentioned, the two categories, one is the color coordinates and one is the conditional coordinates. How do you represent these two different categories? So that's a great question, and in a couple of slides I'll get to it. Exactly. So the data representation, I'll elaborate in a little bit, but loosely, the data representation is for a neural network, is in the weights of each of those arrows that connect the neurons. That's where the representation is. So I'll show to really clarify that example of what that means. The Cartesian versus polar coordinates is just a visual, a very simple visualization of the concept. But you want to be able to represent the data in an arbitrary way where there's no limits to the representation. It could be highly nonlinear, highly complex. Any other questions?

Yep, I believe you touched on it earlier. Generally speaking, I guess in our current state, when we talk about machine learning or AI, it's simply statistical models that are able to recognize trends or patterns or things of that nature where they're not necessarily thinking, but simply recognizing. And so I'm a little confused about how the current, I guess, system differs.

From deep learning and whether you think.

That there is the possibility of transitioning.

From recognizing to actually thinking. So I have a couple of slides almost asking this question because there's no good answers. But one could argue, and I think somebody in the last class brought up that is machine learning just pattern recognition? It's possible that reasoning thinking is just pattern recognition. And I'll describe sort of an intuition behind that. So we tend to respect thinking a lot because we've recently, as human beings, learned to do it in our evolutionary time. We think that it's somehow special from, for example, perception. We've had visual perception for several orders of magnitude longer in our evolution as a living species. We've started to learn to reason, I think about 100,000 years ago. So we think it's somehow special from the same kind of mechanism we use for seeing things. Perhaps it's exactly the same thing. So perception is pattern recognition. Perhaps reasoning is just a few more layers of that.

That's the hope.

It's an open question. The concept of neural network is itself is not very new.

So what, is there any technical innovation, breakthrough to.

To expand the use of neural network?

I would think more.

It's just, you know, increase the result of computational power. Yes, that's a great question there. There's been very few breakthroughs in neural networks since through the AI winters that we discussed through A lot of excitement in spurts. And even recently there's been very few algorithmic innovations. The big gains came from compute, so improvements in GPU and better faster computers. You can't underestimate the power of community. So the ability to share code and the Internet, ability to communicate together through the Internet and work on code together and then digitization of data. So like ability to have large data sets, easily accessible and downloadable, all of those little things. But I think in terms of the future of deep learning and machine learning, it all rides on compute, I think meaning continued bigger and faster computers. That doesn't necessarily mean Moore's law in making smaller and smaller chips. It means getting clever in different directions, massive parallelization, coming up with ways to do super efficient, power efficient implementations in neural networks and so on. So let me just fly through a few examples of what we can do with machine learning just to give you a flavor. I think in future lectures as possible, we'll discuss with different speakers, different specific applications, really dig into those. So we can, as opposed to working with just images, you can work with videos and segment those. I mentioned image segmentation. We do video segmentation. Go through video segment the different parts of a scene that's useful to a particular application. Here in driving, you can segment the road from cars and vegetation and lane markings. You can also. This is a subtle but important point.

Yes.

Just go back that one slide. How do you see the light? I mean that's such a critical piece of.

The more I listen to you and.

Read your stuff, it seems like this critical. These very small pieces of information that we just, we know are important. Like there is a red light or the light, I have to stop, I have to slow down. How does it filter that out and pick out that?

So this, which is.

It's got to be 100% reliable on that, right?

Oof.

Hard question. So.

The question was, how do you detect the traffic, light and lights? So how do we do it as human beings?

First of all, let's start there. The way we do it is by.

The knowledge we bring to the table. So we know what it means to be on the road. There's a lot of the huge network of knowledge that you come with. And so that makes the perception problem much easier. This is pure perception. You take an image and you separate different parts based purely on tiny patterns of pixels. So first it finds all the edges and it learns that traffic lights have certain kinds of edges around them and then zoom out a little bit. They have a certain collection of edges that make up this black rectangle type shape. So it's all about shapes. It kind of build up knowing the shape, structure of things. But it's a purely perception problem. And one of the things I argue is that if it's purely a perception approach and you bring no knowledge to the table about the physics of the world, the three dimensional physics and the temporal dynamics, that you are not going to be able to successfully achieve near 100% accuracy on some of these systems. So that's exactly the right question is.

For all of these things.

Think about how you as a human being would solve these problems and what is lacking in the machine learning approach, what data is lacking in the machine learning approach in order to achieve the same kind of results, the same kind of reasoning required to, that you would use as a human. So there is also image detection, image detection, which means it's a subtle but important point. The stuff I mentioned before, image classification is given an image of a cat, you find the cat. No, sorry, you don't find the cat. You say this image is of a cat or not. And then detection or localization is when you actually find where in the image. That is that problem is much harder but also doable with machine learning, with deep neural networks. Now, as I said, inputs, outputs can be anything. The input could be video, the output can be video. And you could do anything you want with these videos. You can colorize the video, you can add, take an old black and white film and produce color images. Again, in terms of having an impact in the world using these applications, you have to think this is a cool demonstration, but how well does it actually work in the real world? Translation, whether that's from text to text or image to image, you can translate here dark chocolate from one language to another. It's class global business of artificial intelligence. There's a reference below, there you can go and generate your own text. You can generate the writing of the act of generating handwriting. You can type in some text and given different styles that it learns from other handwriting samples, it can generate any kind of text using handwriting. Again, the input is language. The output is a sequence of writing of pen movements on the screen. You can complete sentences. This is kind of a fun one where if you start.

So you can.

Generate language and you can generate language where you start, you feed the system some input first. So in black there it says life is and then have the neural network complete those sentences. Life is about kids, life is about the weather. There's a lot of knowledge here, I think, being conveyed and you can start to Sense what the meaning of life is. The meaning of life is literary recognition true for us academics or the meaning of life is the tradition of ancient human production also true. But these are all generated by a computer. You can also caption this has been become very popular recently is caption generation given this input is an image. The output is a set of text that captions the content of the image. You find the different objects in the, in the image, that's a perception problem. And once you find the different objects, you, you stitch them together in a sentence that makes sense. You generate a bunch of sentences and classify which sentence is the most likely to fit this image. And you can. So certainly in the. I try to avoid mentioning driving too much because it is my field. That is what I'm excited about. But then the moment I start talking about driving, it'll all be about driving. But I should mention of course that deep learning is critical to driving applications. For both the perception and what is really exciting to us now is the end to end, the end to end approach. So whenever you say end to end in any application, what that means is you start from the very raw inputs that the system gets and you produce the very final output that's expected of the system. So as opposed to in the self driving car case, as opposed to breaking a car down into each individual components of perception, localization, mapping, control, planning, and just taking the whole stack and just ignoring all the super complex problems in the middle, just taking the external scene as input and as output produce steering and acceleration braking commands. And so in this way, taking this input as the image of the external world, in this case in a Tesla, we can generate steering commands for, for the car. Again, input a bunch of numbers that, that's just images. Output a single number that gives you the steering of the, of the car. Okay, so let's step back for a second and think about what can't we do with machine learning. We talk, we talked about you can map numbers to numbers. Let's think about what we can't do. This at the core of artificial intelligence in terms of making an impact on.

This world is robotics.

So what can't we solve in robotics and artificial intelligence with a machine learning approach? And let's break down what artificial intelligence means. Here's a stack. Starting at the very top is the environment, the world that you operate in. There's sensors that sense that world. There is feature extraction and learning from that data. And there's some reasoning, planning and effectors are the ways you manipulate the world. What can't we learn in this way. So we've had a lot of success as Jonathan talked about in the history of AI with formal tasks playing games, solving puzzles. Recently we're having a lot of breakthroughs with medical diagnosis. We're still sorry, we're still struggling but are very excited about in the robotics space with more mundane tasks of walking, of basic perception, of natural language, written and spoken. And then there is the human tasks which are perhaps completely out of reach of this pipeline at the moment is cognition, imagination.

Subjective experience.

So high level reasoning. Not just common sense, but high level human level reasoning. So let's fly through this pipeline. There's sensors, cameras, lidar audio, there's communication that flies through the air or wired or wireless or wired imu measuring the movement of things. So that's the way to think about it. That's the way as human beings and as any kind of system that you design, you measure the world. You don't just get an API to the world, you need to somehow measure aspects of this world. So that's how you get the data. So that's how you convert the world into data you can play with. And once you have the data, this is the representation side. You have to convert that raw data.

Of raw pixels, raw audio, raw lidar data.

You have to convert that into data that's useful for the intelligence system, for the learning system to use to discriminate between one thing and another for vision. That's finding edges, corners, object parts and entire objects. There's the machine learning that I'll talk that I've talked about. There's different kinds of mapping of the representation that you've learned to an actual outputs. There is once you have this so you have this idea of and this.

Goes to maybe a little bit of.

Simon's question is reasoning. This is something that's out of reach of machine learning at the moment. This is going to your question. Then we can build a world class machine learning system for taking an image and classifying that it's a duck. I wonder if this will work.

Wake you up.

So we could take this is well studied, exceptionally well studied problem. We could take audio sample of a duck and tell that it's a duck. In fact, what species of bird. It's incredible how much research there is in bird species classification. And you can look at video and.

We could tell that we can do.

Action recognition that it's swimming, but we can't do with learning now is reason that if it looks like a duck, it swims like a duck and quacks like a duck.

It's very likely to be a duck.

This is the reasoning problem. This is the task that I personally am obsessed with and that I hope that machine learning can close. And then there is the planning action and the effectors. So this is another place where.

Machine.

Learning has not had many strides. There's mechanical issues here that are incredibly difficult. The degrees of freedom with all the actuators involved, with all the. Just the ability to localize every part of yourself in this dynamic space where things are constantly changing, when there's degrees of uncertainty, when there's noise, just that basic problem is exceptionally difficult. So let me just pose this question. We talked about how machine, what machine learning can do with the cats and the duck. We could do that given representation. It could predict what's in the image. But one of the open questions is, and deep learning has been able to do the feature extraction, the representation learning. This is the big breakthrough that everybody's excited about. But can it also reason? These are the open questions. Can it reason? Can it do the planning and action? And as human beings do, can it close the loop entirely from sensors to effectors? So learn not only the brain, but the way you sense the world and the way you affect the world. The thing about that pawn game.

So essentially, does the rule network get punished.

Because it goes off the map?

Is that how it learns?

So the question was about the pawn game.

Thank you. I get to talk to it for a little longer.

It doesn't get punished when it doesn't detect the ball. This is the beautiful thing. It gets punished only at the very end of the game for losing the game and gets rewarded for winning the game. So it knows nothing about that ball and it learns about that ball. That's something you need to really sit and think about.

Because, like, how do.

As human beings, imagine if you're playing with a physical ball, how do you learn what a ball is? You get hurt by it.

You, you like squeeze it, you throw.

It, you feel the dynamics of it.

The physics of it.

And nobody tells you about what a ball is. You're just using the raw sensory input. We take it for granted. And maybe this is what I can end on is this is something Jonathan brought up is we take the simplicity.

Of this task for granted because we've.

Been, We've had eyes, we, broadly speaking, as living species on planet Earth. These eyes have been evolved for 540 million years. So we have 540 million years of data. We've been walking for close to that bipedal Mammals, we have been thinking only very recently, so 100,000 years versus 100 million years. And that's why we can't. Some of these problems that we're trying to solve, you can't take for granted.

How actually difficult they are.

So for example, this is the Marv X paradox that Jonathan brought up is that the easy problems are hard. The things we think are easy, actually really hard. This is state of the art robot on the right playing soccer. And that was a state of the art human on the left playing soccer.

And.

I'll give it a second. The question was, you know, there's a fundamental difference between the way we train neural networks and the way we've trained biological neural networks through evolution by discarding through natural selection a bunch of the neural networks that didn't work so well. So first of all, the process of evolution is, I think, not well understood. Meaning, sorry, the role.

Careful here.

The role of evolution in the evolution of our cognition of our intelligence. I don't know if that's. So this is an open question, so maybe clarify this point is neural networks, artificial neural networks are fixed for the most part in size. This is exactly right. It's like a single human being that gets to learn. We don't have mechanisms of modifying or evolving those neural networks yet, although you could think of researchers as doing exactly that. They're. You have grad students working on different neural networks and the ones that don't do a good job, don't get promoted and get a good. You know, there is a natural selection there, but other than that, it's an open question. It's a fascinating one.

So Lex is going to come back. He's not available next week, but he's going to come back the week after. So we can pick up many of these points here.

Many of these points continue.

Are there any last final takeaways you want to emphasize?

Stay tuned and keep your head up because the future, I believe, is really promising. And the slides will be made available for sure.

I think a lot of the explorations of what it means to build an intelligent machine has been in sci fi movies. We're now beginning to actually make it a reality. This is Space Odyssey. To keep with that theme. In the previous lecture, the we had this is as opposed to the dreamlike monolith view when the astronaut is gazing out into the open sky at the stars. We're going to look at the practice of AI today and how we go. If you're familiar with the movie, when this new technology appeared before our eyes and we're full of excitement how we transfer that into actual practical impact on our lives. To quickly review what we talked about last time, I presented the technology and asked the question of whether this technology merely serves a special purpose to answer specific tasks that can be formalized, or whether it can be, through the process of transferring the knowledge learned on one domain, be generalizable to where an intelligent system that's trained in a small domain can be used to achieve general intelligent tasks like we do as human beings. This is kind of the stack of artificial intelligence going from all the way up to the top of the environment, the world, the sensors, sensor data, the intelligent system, the way it perceives this world. Then once you have this, you convert the world into some numbers, you're able to extract some representation of that world. And, and this is where machine learning starts to come into play. And then there's the part where I will raise it again today is can machine learning be doing the following steps too? That we can do very well as human beings is the reasoning step. You know, you can tell the difference between a cat and a dog, but can you now start to reason about what it means to be alive, what it means to be a cat with living creature, and what it means to be this kind of physical object or this kind of physical object and take what's called common sense, things we take for granted, start to construct models of the world through reasoning? Descartes I think, therefore I am. We want our neural networks to come up with that on their own. And once you do that action, you'll go right back into the world and you start acting in that world. So the question is, can machine learning, can this be learned from data, or do experts need to encode the knowledge of reasoning, the knowledge of actions, the set of actions? That's kind of the question, open questions I raised. It continues throughout the talk today. And so as we start to think about how artificial intelligence, especially machine learning, realizes itself through robotics, gets to impact the world, we start thinking about what are the easy problems, what are the hard problems? And it seems to us that vision and movement, walking is easy because we've been doing it for millions of years, hundreds of millions of years. And thinking is hard, reasoning is hard. I propose to you that it's perhaps because we've only been doing it for a short time and so think we're quite special because we're able to think.

So we have to kind of question.

Of what is easy and what is hard. Because when we start to develop some of these systems oh man.

What?

You start to realize that all of these problems are equally hard. So the problem of walking that we take for granted, the actuation and the physical, the ability to recognize where you are in the physical space, to sense the world around you, to deal, deal with the uncertainty of the perception problem. And then so all of these robots, by the way, this is for the most recent DARPA challenge, which MIT was also part of.

And so what are these robots doing?

They don't have any. They only have sparse communication with human beings on the periphery. So most of the stuff they have to do autonomously, like get inside a car, this is an MIT robot. Unfortunately, they have to get in the car and the hardest task, they have.

To get out of the car.

That's walking. So this kind of raises to you a very real aspect here. You want to build applications that actually work in the real world. And that's the first challenge and opportunity here that many of the technologies we talked about currently crumble under the reality of our world when we transfer them from a small data set in the lab to the real world. For the computer vision is perhaps one of the best illustrations of this computer vision is the task, as we talked about, of interpreting images. And so when you. There's been a lot of great accomplishments on interpreting images. Cats versus dogs. Now when you try to create a system like the Tesla vehicle that I've often, that we work with and I always talk about is it's a vision based robot, right?

It has radar for basic obstacle avoidance.

But most of the understanding of the world comes from a single monocular camera. Now they've expanded the number of cameras, but for the most time there's been 100,000 vehicles driving on the roads today with a single, essentially a single webcam. So when you start to do that, you have to perform all of these. Extraction of texture, color, optical flow, so the movement through time, temporal dynamics of the images. You have to construct these patterns, construct the understanding of objects and entities and how they interact. And from that you have to act in this world. And that's all based on this computer vision system. So it's no longer cats versus dogs, it's detection of pedestrians, where the wrong classification, the wrong detection is the difference between life and death. So let's look at cats, where things are a little more comfortable. So computer vision and I would like to illustrate to you why this is such a hard task.

We talked about.

We've been doing it for 500 million years, so we think it's easy. Computer vision is actually incredible. So all you're getting with your human eyes is you're getting essentially pixels in. There's light coming into your eyes, and all you're getting is the reflection from the different surfaces in here of light. And there's perception, there's sensors inside your eyes converting that into numbers. It's really very similar to this numbers, in the case of what we use with computers, RGB images, or the individual pixels that are numbers from 0 to 255, so 256 possible numbers, and there's just a bunch of them. And that's all we get. We get a collection of numbers where they're spatially connected. Ones that are close together are part of the same object. So cat pixels are all connected together. That's the only thing we have to help us. But the rest of it is just numbers, intensity numbers, and we have to use those numbers to classify what's in the image. And if you really think about it, this is a really difficult task. All you get is these numbers. How the heck are you supposed to form a model of the world with which you can detect pedestrians with really 99.9999% accuracy? Because these pedestrians, or these cars and cyclists in the car context, or any kind of applications you're looking at, even if your job is on the factory floor to detect the defective gummy bears that are flying past at like 100 miles an hour, your task is, you don't want that bad gummy bear to get by, that your product and the brand will be damaged. However serious or not serious your application is what you have to be. You have to have a computer vision system that deals with all of these aspects. Viewpoint variation, scale variation, no matter the size of the object is still the same object. No matter the viewpoint from which area you look at, that object is still the same object. The lighting that moves. We have lighting consistently here because we're indoors. When you're outdoors or you're moving, the scene is moving. The lighting. The complexity of the lighting variations is incredible, from the illumination to just the movement of the different objects in the scene.

Now, we've had these conversations. I think about this every time I drive. I think about you and this point and how hard it is to see these things. And particularly when I'm driving at night, and particularly when it's twilight and the light is changing, I think almost every time I drive, there's one or two things that I see that are really, that I'm drawing on, like 200 million years in order to be able to Figure out it's a guy who's opened his car door and I can't see him, but I can just see the light doesn't look quite right on that side of the road. And somehow I know in my mind it's a person. But it seems like an almost impossible problem for the machines to get right with sufficient accuracy.

I will argue that the pure perception task is too hard, that you come to the table as human beings with all this huge amount of knowledge that you're not actually interpreting all the complex lighting variations that you're seeing. You actually know enough about the world, you enough about your commute home, enough about the way, the kinds of things you would see in this world, about Boston, about the way pedestrians move, the certain light of day. You bring all that to the table. That makes the perception task doable. And that's one of the big missing pieces in the technology, as I'll talk about. That's the open problem of machine learning, is how to bring all that knowledge, first of all build that knowledge and then bring that knowledge to the table as opposed to starting from scratch every time. And so cats, I promise, cats. Okay, so to me, occlusion. For most of the computer vision community, this is one of the biggest challenges and it really highlights how far we are from being able to reason about this world. Occlusions are when what an occlusion is, is when the objects you're trying to detect something about, classify the object, detect the object, the object is blocked partially by another object in front of them. This is something you think is trivial. Perhaps you don't even really think about it because we reason in a three dimensional way. But the occlusion aspect makes perception incredibly difficult. So we have to design, think about this. So this image is converted into numbers and we for the task of detecting is there a cat in this image, yes or no? You have to be able to reason about this image with that object in the scene. Most of us are able to very easily detect that there's a cat in this image. We're able to detect that there is a cat in this image. Now think about this. There's a single eye and there's an ear. So you have to think about what is it part of our brain that allows us to understand, to suppose that with some high degree of accuracy that there's a cat here in this picture. I mean, the degree of occlusion here is immense. So I promise. So this is for most of you, some of you will think this is in fact a monkey eating A banana. But I would venture to say that most of us are able to tell it's nevertheless a cat. You watch this for hours. And so let me give you another. This is kind of a paper that's often cited or a set of papers that illustrate how difficult computer vision is, how thin the line that we're walking with all of these impressive results that we've been able to show recently in the machine learning community, in this case for deep neural networks are easily fooled paper. A seminal paper at this point shows that when you apply Network trained on ImageNet, so basically on detecting cats versus dogs or different categories inside images, you can find an arbitrary number of images that look like noise up in the top row, where the algorithm used to classify Those images in ImageNet of cat versus dog is able to confidently say with 99.6% accuracy or above that it's seeing a robin or a cheetah or an armadillo or a panda in that noise. So it's confidently saying given this noise that that's obviously a robin. So you have to realize that the kind of this is patterns, the kind of processes it's using to understand what's contained in the image is purely a collection of patterns that it has been able to extract from other images that has been annotated by humans. And that perhaps is very limiting to trying to create a system that that's able to operate in the real world. This is a very. So this is a very clean illustration of that concept. And the same you can confidently predict in those images below, where there are strong patterns, it's not even noise, strong patterns that have nothing to do with the entities being detected. Again, confidently, that same algorithm is able to see a penguin, a starfish, a baseball and a guitar in that noise. And more serious, for people designing robots like myself on the sensor side, you can flip that and say I can take a image and I can distort it with some very little amount of noise. And if that noise is applied to the image, I can completely change the confident prediction about what's in that image. So to explain what's being shown. So on the left, in the column on the left and again here, what's the same kind of neural network is able to predict accurately, confidently that there is a dog in that image. But if we apply just a little bit of noise to that image to produce that image imperceptible to our human eyes, the difference between those two, the same algorithm is saying that there is confidently an ostrich in that image. So another thing to really think about that noise can have such a significant impact on the prediction of these algorithms. This is really, really, quite honestly, out of all the things I'll say today, and I'm aware of one of the biggest challenges of machine learning being applied in the real world is robustness, how much noise can you add into the system before everything falls apart? So how do you validate sensors? So say a car company has to produce a vehicle and it has sensors in that vehicle. How do you know that those sensors will not start generating slight noise due to interference of various kinds? And because of that noise, instead of seeing a pedestrian, it will see nothing or the opposite, you'll see pedestrians everywhere. So, of course, the most dangerous is when it will not see an object and collide with it. In the case of cars, there's also spoofing, which a lot of people, as always with security people are really concerned about, and perhaps people here are really concerned about this issue. I think this is a really important issue. But because you can apply noise and convince the system that you're seeing an ostrich when there's in fact no ostrich, you can do the same thing in an attacking way. So you can attack the senses of a car and make it believe, like with lidar spoofing. So spoof lidar or radar or ultrasonic sensors to believe that you're seeing pedestrians when they're not there. And the opposite, to hide pedestrians, make pedestrians invisible to the sensor when they're in fact there. So whenever you have intelligent systems operating in this world, they become susceptible to the fact that everything, so much of the work is done in software and based on sensors. So at any point in the chain, if there's a failure, you have to be able to detect that failure. And right now, we have no mechanisms for automatically detecting that failure. So on the data side, so one challenge is that we're constantly dealing with.

Is.

That we, the algorithms in machine learning, algorithms that we're using need labeled.

Data, and we have very little labeled data.

Labeled data, again is when you have pairs of input data and the ground truth, the true label annotation class, that that image belongs to or concept. And it doesn't have to be an image, it could be any source of data. It's a really costly process to do so. Because it's so costly, we rely. Every breakthrough we've had so far relies on that labeled data. And because of its cost, we don't have much of it. So all the problems that come from data can either be solved by having a lot more of this data, which I believe is, most people believe is too challenging, is too challenging to have human beings annotate huge amounts of data. Or we have to develop algorithms that are able to do something with the unlabeled data. It's the unsupervised, semi supervised, sparsely supervised reinforcement learning as we talked about last time, I'll mention again here. So one way you understand something about data when you don't have labels is you reason about it. All you're given is a few facts. When you're a baby, your parents give you a few facts and you go into this world with those facts and you grow your knowledge graph your knowledge base your understanding of the world from those few facts. We don't have a good method of doing that in an automated, unrestricted way. The inefficiency of our learners, the machine learning algorithms I've talked about, neural networks need a lot of examples of every single concept that they're given in order to learn anything about them. Thousands, tens of thousands of cats are needed to understand what the spatial patterns at every level. The representation of a cat, the visual representation of a cat. We can't do anything with a single example. There's a few approaches, but nothing quite.

Robust yet.

And we haven't come up with a way. This is also possible to make annotation, this labeling process somehow be very cheap. So leveraging this is something being called human computation. That term has fallen out of favor a little bit. One of my big passions is human computation is using something about our behavior, something about what we do in this world online or in the real world to annotate data automatically. So for example, as you drive, which.

Is what we do, everybody has to.

Drive and we can collect data about you driving in order to train self driving vehicles to drive. And that's a free annotation. So here are the annotated data sets we have the supervised learning data sets. There's many, but these are some of the more famous ones. From the toy data sets of mnist to the large, broad, arbitrary categories of images data sets, which is what imagenet is. And there's in healthcare, there's in audio, there's in video, there's, you know, there's a huge number of data sets now, but each one of them is usually in a scale of hundred, thousands, millions, tens of millions, not billions or trillions, which is what we need to create systems that operate in the real world. And again, these are the kinds of machine learning algorithms we have. There's five Listed here, the teachers on the left, is what is. What is the input to the system that requires to train it from the supervised learning. At the very top is where we have all of our successes. And everything else is where the promise lies. The semi supervised, the reinforcement, or the fully unsupervised learning where the input from the human is very minimal. And another way to think about this, so every, whenever you think about machine learning today, whenever somebody talks about machine learning, what they're talking about is systems that memorize, that memorize patterns. And so this is one of the big criticisms of the current machine learning approaches, where all they're doing is you're providing they're only as good as the human annotated data that they're provided. We don't have mechanisms for actually understanding. You can pause and think about this. In order to create an intelligent system, it shouldn't just memorize, it should understand the representations inside that data in order to operate in that world. And that's the open question. One of them, and one of the challenges and opportunities for machine learning researchers today is to extend machine learning from memorization to understanding this is that duck the reasoning. If you get information from the perception systems that it looks like a duck, from the audio processing that it quacks like a duck, and then from video classification that it, the activity recognition that it swims like a duck, the reasoning step is how to connect those facts to then say that it is in fact a duck. Okay, so that's on the algorithm side and the data side. Now this is one of the reasons compute computational power, computational hardware that is at the core of the success of machine learning. So our algorithms have been the same since the 60s, since the 80s, 90s, depending on how you're counting. The big breakthroughs came in compute. So there's Moore's Law. Most of you know, the way the CPU side of our computers works for a single CPU is that it's for the most part executing a single action at a time in a sequence. So sequential, very different from our brain, which is massively parallelized system. So because it's sequential, the clock speed matters because that's how fast essentially those instructions are able to be executed. And so we're leveling off. Physics is stopping us from continuing Moore's Law. Intel AMD are aggressively pushing this Moore's Law forward. But and there's some promise that it'll actually continue for another 10 or 15 years. Then there's another form of parallelism. Massive parallelism is the gpu. And this is this is essential for neural networks. This is essential to the success, recent success of neural networks is the ability to utilize these inherently parallel architectures of graphics processing units, GPUs, the same thing used for video games. This is the, this is the reason Nvidia stock is doing extremely well is GPUs. So it's parallelism of basic computational processes that make machine learning work on the GPU. One of the limitations of GPUs, one of the challenges is in bringing them to, in scaling and bringing them into real world applications is power usage, is power consumption. And so there is a lot of specialized chips, specialized just from the neural network architectures coming out from Google with their tensor processing unit, from IBM, intel and so on. It's unclear how far this goes. So this is sort of the direction of trying to design an electronic brain. So it has the efficiency, our human brain is exceptionally efficient and at running the neural networks in our heads, orders of magnitude more efficient than our computers are. And this is trying to design systems that are able to go towards that efficiency. Why do you care about efficiency? For several reasons. One, of course, as I'm sure we'll talk about throughout this class, is about the thing in our smartphones, battery usage. And this is the big one community. I think.

I think it could be.

Attributed to the big breakthroughs in machine learning recently, in the last decade is the, you know, compute is important, algorithm development is important, but it's the community of nerds. Global. This is global artificial intelligence. And I will show in several ways why global is essential. Here is tens of hundreds of thousands, millions of programmers, mechanical engineers building robots, building intelligent systems, building machine learning algorithms. The exciting nature of the growth of the community perhaps is the key for the future to unlocking the power of machine learning. So this is just one example. GitHub is a repository for code. And, and this is showing on the Y axis at the bottom is 2008 when GitHub first opened. This is going up to 2012. Quick near exponential growth of the number of users participating and the number of repositories. So these are standalone, unique projects that are being hosted on GitHub. So this is one example. I'll show you about this competition that we're recently running and then I'll challenge people here to participate in this competition, if you dare. So this is a chance for you to build a neural network in your browser. So you can do this on your phone later tonight. Of course, on your phone you can specify various parameters of the neural network, specify different numbers of Layers and the depth, the depth of the network, the number of neurons in the network, the type of layers, and it's pretty self explanatory, super easy in terms of just tweaking little things. And remember, machine learning to a large part is an art at this point. It's more perhaps than even, you know, more than a well understood, theoretically bounded science, which is one of the challenges, but it's also an opportunity. Deep traffic is a chance. So we've all been stuck in traffic.

There you go.

Americans spend 8 billion hours stuck in traffic every year. That's our pitch for this competition. So deep neural networks can help. And so you have a neural network that drives that little car with an MIT logo, red one, on this highway and tries to weave in and out of traffic to get to his destination and trying to achieve a speed of 80 miles an hour, which is the speed limit, which is the physical speed limit of the car. Of course, the actual speed limit of the road is 65 miles an hour, but we don't care about that. We just want to get to work as quickly as possible at home. So what the basic structure of this game is, and I want to explain this game a little bit and then tell you how incredibly popular it's gotten and how incredibly powerful. The networks that people built from all over the world, the community that's built of this over a single month is incredible. And this happens for thousands of projects out there now. Another challenging opportunity. Okay, so you may have seen this. This is kind of ethics. Most engineers, most, I personally don't like, I love philosophy. But this kind of construction of ethics that's often presented here is one that is not usually concerned to engineering. So what is this question? You know, when you have a car and you have a bunch of pedestrians, do you hit the larger group of pedestrians or the smaller group of pedestrians? Do you avoid the group of pedestrians, put yourself into danger? These kinds of ethical questions of an intelligent system. It's a very interesting question. It's one that we can debate. And there's really no good answer, quite honestly. But it's a problem that both humans and machines struggle with. And so it's not interesting on the engineering side. We're interested with problems that we can solve on the engineering side. So the kind of problem that I'm obsessed with and very interested in is, is the real world problem of controlling.

A vehicle through this space.

So it happens in a few seconds here. So this is a Manhattan, New York intersection, right? This is pedestrians walking perfectly legally.

I think they have A green light.

Of course, there's a lot of jaywalking too as well. Well, this car just flies. It's not part of the point, but yes, exactly. There's an ambulance and so there's another car that starts making a left turn in a little bit. I may have missed it. Hopefully not. So yeah, and then there's another car after that too. That just illustrates when you design an algorithm that's supposed to move through the space, like, watch this car, the aggression it shows. Now this isn't a trivial example for those that try to build robots. This is, this is the real question is how do you design a system that's able. So you have to think, you have to put reward functions, objective functions, utility functions under which it performs the planning. So a car like that has several thousand candidate trajectories. You can take through that intersection. It can take a trajectory where it speeds up to 60 miles an hour and doesn't stop and just swerves and hits everything. Okay, that's a bad trajectory, right? Then there's a trajectory which most companies take, which most the Google self driving car and every company that is concerned about PR is. Whenever there's any kind of obstacle, any kind of risk that's at all reasonable that you can maybe even touch an obstacle, then you're not going to take that trajectory. So what that means is you're going to navigate through this intersection at 10 miles an hour and let people abuse you by walking in front of you because they know you're not going to stop. And so in the middle, there is hundreds of thousands of trajectories that are ethically questionable in the sense that you're putting other human beings at risk in order to safely and successfully navigate through an intersection. And the design of those objective functions is the kind of question you have to ask for intelligence systems, for cars. There's no grandma and a few children. You have to choose who gets to die. Very difficult problems, of course, but the problem of one I'm very interested in the streets of Boston, streets of New York, is how to gently nudge yourself through a crowd of pedestrians in the way we all actually do when we drive in New York, in order to be able to safely navigate these environments. And these questions come up in health care, these questions come up in factories, in robots, in armed and humanoid robots that operate with other human beings. And that's one of the big challenges, another sort of fun illustration that folks at OpenAI use often to illustrate. Well, let me just pause for a second. The Gamified version of this. There's a game called Coast Runners and you're racing against other boats along this track. And your job is there's your score here at the bottom left, number of laps, your time, and you're trying to get to the destination as quickly as possible while also collecting funky little things like these green little things along the way. Ok, so what they've done is the build end to end system, the one, the general purpose one that we talked about last time that learns how to navigate successfully through the space. So you're trying to maximize the reward. And what this boat learns to do is instead of finishing the race, it learns to find a loop where it can keep going around and around collecting those green dots. And it learns the fact that they regenerate with time. So it learns to maximize the score by going around and around. Now these are the kinds of things, this is the big challenge of reward functions, of designing systems, of designing what you want your system to achieve is not only is it difficult to. The ethical questions are difficult, but just avoiding the pitfalls of local optima, of figuring out something really good that happens in the short term. The greedy. What are those psychology experiments where the kid eats the marshmallow and can't wait for, you know, can't delay gratification. This kind of. The idea of delayed gratification in the case of designing intelligent systems is a huge, actual serious problem. And this is a good illustration of that. So we flew through a few concepts here. Is there any questions about some of the compute. And it's a very interesting question. It's one that we can debate and there's really no good answer, quite honestly. But it's a problem that both humans and machines struggle with. And so it's not interesting on the engineering side. We're interested with problems that we can solve on the engineering side. So the kind of problem that I'm obsessed with and very interested in is the real world problem of controlling a.

Vehicle through this space.

So it happens in a few seconds here. So this is a Manhattan New York intersection, right? This is pedestrians walking perfectly legally. I think they have a green light. Of course there's a lot of jaywalking too as well. Well, this car just flies. It's not part of the point, but yes, exactly. There's an ambulance and so there's another car that starts making a left turn in a little bit. May have missed it. Hopefully not. So yeah, and then there's another car after that too. That just illustrates when you design an algorithm that's Supposed to move through the space like, watch this car, the aggression it shows. Now this isn't a trivial example for those that try to build robots. This is, this is the real question is how do you design a system that's able. So you have to think, you have to put reward functions, objective functions, utility functions under which it performs the planning. So a car like that has several thousand candidate trajectories. It can take through that intersection. It can take a trajectory where it speeds up to 60 miles an hour and doesn't stop and just swerves and hits everything. Okay, that's a bad trajectory, right? Then there's a trajectory which most companies take, which most the Google self driving car and every company that is concerned about PR is whenever there's any kind of obstacle, any kind of risk that's at all reasonable that you can maybe even touch an obstacle, then you're not going to take that trajectory. So what that means is you're going to navigate through this intersection at 10 miles an hour and let people abuse you by walking in front of you because they know you're not going to stop. And so in the middle, there is hundreds, thousands of trajectories that are ethically questionable in the sense that you're putting other human beings at risk in order to safely and successfully navigate through an intersection. And the design of those objective functions is the kind of question you have to ask for intelligence systems. For cars, there's no grandma and a few children. You have to choose who gets to die. Very difficult problems, of course, but the problem of one I'm very interested in the streets of Boston, streets of New York, is how to gently nudge yourself through a crowd of pedestrians in the way we all actually do when we drive in New York, in order to be able to safely navigate these environments. And these questions come up in health care, these questions come up in factory, in robots, in armed and humanoid robots that operate with other human beings. And that's one of the big challenges. Another sort of fun illustration that folks at OpenAI use often to illustrate. Well, let me just pause for a second. The gamified version of this. There's a game called Coast Runners and you're racing against other boats along this track. And your job is there's your score here at the bottom left, number of laps, your time, and you're trying to get to the destination as quickly as possible while also collecting funky little things like these green little things along the way. Okay? So what they've done is a build, end to end system. The one the general purpose one that we talked about last time, that learns. Oops. That learns how to navigate successfully through the space. So you're trying to maximize the reward. And what this boat learns to do is instead of finishing the race, it learns to find a loop where it can keep going around and around collecting those green dots. And it learns the fact that they regenerate with time. So it learns to maximize the score by going round and round. Now, these are the kinds of things, this is the big challenge of reward functions, of designing systems, of designing what you want your system to achieve is not only is it difficult to. The ethical questions are difficult, but just avoiding the pitfalls of local optima, of figuring out something really good that happens in the short term. The greedy. What are those psychology experiments with the kid eats the marshmallow and can't wait for, you know, can't delay gratification. This kind of. The idea of delayed gratification in the case of designing intelligent systems is a huge, actual serious problem. And this is a good illustration of that.

So.

We flew through a few concepts here. Is there any, is there any questions about some of the compute and the algorithm side we talked about today? Yes. So the question was, yeah, you highlighted some of the limitations of machine, computer vision algorithms, machine learning algorithms, but you haven't highlighted some of the limitations of human beings. And if you put those in a column and you compare those, is are machines doing better overall or is there any kind of way to compare those? I mean, there is actually interesting work on imagenet. So imagenet is this categorization task of where you have to clarify, classify images and you can ask the question when I present you images of cats and dogs, where are machines better than humans and when are they not? So you can compare when machines do better, what are the fail points and what are the fail points for humans? And there's a lot of interesting visual perception questions there. But I think overall it's certainly true that machines fail differently than human beings. But in order to make an artificial intelligence system that's usable and could make you a lot of money and people would want to use it has to be better for that particular task in every single way. In order for you to want to use a system, it has to be superior to human performance and usually far superior to human performance. So on the philosophical level, it's an interesting thing to compare what are we good at, what are not? But if you're using Amazon Echo, your voice recognition, or any kind of natural language chatbots or a car, you're not going to be. Well, this car is not so good with pedestrians, but I appreciate the fact that it can stay in the lane. Fortunately, you have a very high standard for every single thing that you're good at and it has to be superior to that. I think maybe that's unfair to the robots. I'm more of the nerd that makes the technology happen, but it's certainly on the self driving car aspect.

Policy is probably the biggest challenge and.

I don't think there's good answers there. Some of those ethical questions that come.

Up, it feels like. So we work a lot with Tesla.

So I'm driving a Tesla around every day and we're playing around with it and studying human behavior inside Teslas. And it seems like there's so much hunger amongst the media to jump on something and it feels like a very shaky PR terrain, a very shaky policy terrain. We're all walking because we have no idea how we coexist with intelligence systems. And then of course government is nervous because how do we regulate this shaky terrain? Everybody's nervous and excited. So I'm not sure there's no grass.

That's a perfect transition point if that's okay. That same kind of question to Jason in a moment. Thanks a lot, Lex for another great session.