2D Image analysis and Pitfalls | Radu Rusu

So I'm here to talk about something a little bit different. I've been looking at the other lectures that most people gave you, and I think this is going to be a little bit of a different flavor. So as your colleague mentioned, I come from a research background. I guess I'm an unusual CEO. Most people actually sort of get their hands dirty much, much, much faster when they're in their 20s, like you guys hopefully, which is great. I'm in my 30s, old, I know. But I think I picked up some things along the way that potentially helped us in shaping, I guess, what we call the next internet. And I'll talk to you about it in the next presentation. So here's one of the projects that I've been actually working on for quite some time at a company called Willow Garage. Who has heard of Willow Garage here? Perfect. Perfect audience. So what we have here is a robot. So we have a robot that costs about $400,000. That you can control by, you know, using web interfaces and a bunch of different types of software. And in this application, what it does, it's, you know, if you're a lazy engineer or whatever, researcher, you're going to order beer from your office, and you have this robot going around the building, searching for refrigerators, finding one, opening it, figuring out if what you ordered is what's available in the fridge, picking up the beer bottles, bringing them to your office. How cool is that, right? Super cool. Yeah. Replace beer with the beverage of your choice. I mean, you can imagine you can have soft drinks or anything like that. So this is a project that we embarked on at this company called Willow Garage, which I'm going to talk about more during my presentation. The goal of what you see here is pretty much to create personal robots, right? And as you'll see in our paradigm of where computers are going, this is the next computer pretty much. It's a computer that's not only mobile because you can move it, but it moves by itself. So this is what I would like to have in my lifetime, to be honest. Obviously in a much nicer form and shape and doing more interesting things. So as I mentioned, we're looking at computing a little bit differently than others. We're saying, hey, we got mainframes, we got PCs. All of a sudden we sort of redefine the idea of mobile, right? Because it's not like I get my laptop and I move it with me. I pretty much have it all the time in my pocket. And that's when we brought in mobile smartphones and tablets. Wearables are coming next. You can't escape them. You should probably embrace them as soon as possible. Forget about Google Glass. OK, yeah, that was a failure. So what? There's more to come. So this is clearly something that people want. And again, in our perspective, we're going to redefine mobile again, re-redefine if you want. And personal robots are coming next. And so you have different types of technologies that are needed to build these kind of computers, and they will basically have different types of sensors and actuators and so on, as you see here. So are we ready for these kind of robots, right? Like, do you think we have the technology? Can we just make them happen, like, tomorrow? Yes, no, I don't know? Yes? Yes. Okay, some people say yes. Not by a long shot, okay? And here's why. It took us about, let's say, 6 months to put together the software packages that were required for the video that you just saw. Obviously, everything was hacked together in a week. It was a hackathon. But all those individual building blocks were built over the course of many months. Afterwards, we tried to take some of those modules and reuse them in other applications. Failure. We couldn't do that. So the conclusion that we had was we need to build something a bit more generalized. You can't always just have one application in mind, build these modules and expect they're going to rework in other applications. So the conclusion that we had is we need more data. We need more robots. So anyway, we think that robots are stupid right now. They're not doing anything interesting. You have to program them by hand, and it's quite tedious. So because they're stupid, consumers won't buy them. Why would I buy a robot that doesn't do anything for me? So therefore, we don't really have enough robots in the field, and we don't have enough data. So it's kind of this vicious circle, this robot chicken and egg problem, as we call it, They would just can't break, can't escape. If we had a lot of really good data, we could train a lot of different classifiers and build really complicated software that would probably make robots a bit more intelligent, but we don't, as you'll see. So don't we already have— people ask me, well, don't we already have a lot of pictures on the internet? What do you mean we don't have enough data? There's so many snaps that everybody's taking all the time. Why can't we just use those to train robots and make— Yeah. get smarter things done. And we also have stuff like deep learning these days. You guys know what deep learning is? Have you heard of deep learning? Deep learning. OK. So deep learning, I'll explain a little bit better, is pretty much a new concept that appeared in the AI machine learning community where you pretty much take neural networks, make them a bit fancier, and use them for classifying things or for regression. So the truth is that we are at a very interesting crossroad right now where we've been fooling ourselves that just by throwing more data, like all these images, into some black box that we call, again, deep learning neural networks and whatnot, we're going to solve a very complicated problem. The reality is a bit different. First of all, the biggest problem that we have is that we're using the wrong kind of data. So we have 2D images, but have 2D images been meant to be used for these kind of applications? Okay? So take the best sort of like approaches on the planet. Go to like the top conferences in the field. Go to Google, to Facebook, to whatever company is kind of like kicks butt in this field, right, in image processing and recognition, and you will see that their products are not that great, right? And for a good reason. It's just very, very difficult to understand semantics. I'll give you a very brief example. Where just because your picture is underexposed or overexposed, you're not going to get any feature matches. You're not going to be able to recognize anything. That's a very basic problem. And even when you do, are you matching an object with an object or an object with a picture of the object? It's just very difficult. Imagine that robot that I mentioned earlier. You're programming it to bring you beer. It brings you an empty mug. That's a failure. You don't want that. So more examples, like if you're trying to recognize things in the real world. You got a camera, you're moving it around, and you should have these kind of boxes always pop up in the right places. Like a mug should always be a mug. But as you can see, as you're moving around from different angles, mugs don't appear as mugs. And you get a lot of false positives in the background and so on. So negatives and false positives are killing recognition. And I think the generalized recognition has really poor results right now. Okay. And no offense, but these are like state-of-the-art results in face recognition. This is the best that we can do right now with all the classifiers that we're training and so forth. So what about deep learning? This really cool thing that popped up a few years ago and everybody's like crazy of money being invested and— well, look, it's machine learning and this has been around for many, many years. There's many flavors of machine learning and every 5 years or 10 years, You get someone kind of looking at something that's been done before, sort of renaming it, making it a little bit better incrementally, and then calling it something else and going almost viral in the community. Like, oh, this is the next best thing. We're going to reinvent the human brain. This has happened many, many, many times. So the problem with these approaches, and you see again lots of companies building this kind of stuff, is that you can recognize things that are very domain-specific. So for example, if you're trying to recognize cats and you're only training those classifiers for cat recognition, you're probably going to do OK after a while. There's some negatives and whatnot here, or false positives, sorry. But it's kind of OK. But how do you take these and apply it to everything else? I want to be able to know what's happening in the world at all times. How do you do that? So obviously entrepreneurs, and for a good reason, they're going out and they're saying, no, no, no, now I can do cats. If you just give me a lot of money, I will be able to generalize this and we're going to do very, very well in everything else, in all the other domains. And this is great. I mean, you guys should do that totally. If someone gives you money for this, that's fantastic. You're probably not going to solve it, but you're probably going to do a little bit better incrementally and so on and move in the right direction. Really, The way VCs should be thinking about this is like, well, let's try to do more than just one domain-specific problem. Let's try to solve like 4, 5, 6, 10, and then we see that you have something that can generalize, and then we should give you money and you should start a business, because otherwise you might fail. Well, the good news is that you might get acquired, right, because a lot of companies again are just trying really hard to break through in this domain. So in my opinion, in our opinion actually at Fusion Open Perception, There's problems that you're never going to be able to solve in an intelligent manner with 2D images. Just because we can't solve them as people. I mean, a lot of times we're thinking, well, yeah, that looks real. That's true because from that angle it looks perfect, but it's not. So we have this thing called experience and so on and associative memory which helps us disambiguate these problems. So as I mentioned, it's very, very, very hard, almost impossible to think that the current approaches that we have will just generalize. And will work for everything else out there. OK. So as I mentioned, the kind of things that I want in my lifetime are at least things like this. More examples with this robot called the PR2. This was big a few years ago, a couple of years ago, like on CNN and all other news media outlets. So what they're doing right now— and bear in mind, there's a green screen behind it, so we're trying to simplify the problem. We're folding towels. We're trying to sort socks in this case. And this is how the robots do it. This is probably the best approach that you can think of for doing this kind of thing. I mean, if you have a better solution, we would love to hear it. But this is state of the art. What you see on the left are, again, like this— so this application of folding towels initially took like 20 minutes per towel. That's kind of inefficient, right? Well, then we went back and optimized it. And everybody was working really hard. It went down to like 3 minutes, right? Still like a lot. Like imagine having to fall like 20 tiles, right? So by the time we get to seconds and then we actually bring in the real world, right, which is no more green screens, things are going to be really, really complicated. So am I really getting that right now? No. This is really what I'm getting. This is state of the art in recognition in 2D, okay? Again, no offense meant. So I think it's safe to conclude that these biologically inspired approaches are not usually the best idea. And here's a very, very good example. We cannot name any mammals or anything like that out there that have a wheel, right? Yet, as a mode of locomotion, yet we as inventors, engineers, and whatnot create this wonderful thing, this circle, right? the wheel that right now, I mean, can you imagine our society functioning without a wheel? It's so efficient. Look around you. Everything's sort of circular and so on. It works really well. So this is an example where you should probably try to think a little bit outside of the box for some of these really hard problems and not just say, well, because people do it— go ahead. I just have to say, it's kind of tangential, but don't veto me. Right. So don't always say, hey, well, we have— people can— we have like 2D cameras basically here, so we can do it, therefore computers should do it because the Google brain or whatever is like, I don't know how much more complicated than our own. But it's not necessarily the best way to go about it. So you should always try to think about some other things. Thank you.