Radu Rusu argues that 2D images are too limited for robotics and AI. Manual annotation doesn't scale, and photos and video are static snapshots of reality. He advocates for 3D sensing, sonar, and other depth technologies to give robots richer, actionable data.
Chapters
So what are the solutions? Can anyone think of what can we do? We got a lot of data. It's really not great. OK, you were first. I don't know, but I just have an idea. Maybe it's multiple different kinds of sensors using things like sonar and not just visual information.
Go on Mechanical Turk and Fiverr and freelance or just hire a bunch of people to sift through this data effectively in ways that a computer can't.
Great.
Anyone else?
Try 3D instead of 2D. Okay. Okay. So that's pretty much what— the direction that we should be going in, and I'll explain that in a second. Just to answer the Mechanical Turk argument, so we're already doing that. In fact, at scale. The sort of the kitten recognition problem, you know, finding cats in images is actually done using immense, like massive datasets that people manually label, and then they use that You know, to solve the problem. And the issue is that we just won't be able to scale. And I mean, who wants to live in a world where half of the planet is kind of like annotating things for the other half? It's just not cool, right? So we've got to move forward. So the solution is obviously to think about other types of data, okay? So I'll revisit the problem of robotics. It's not just we don't have enough data, we don't have enough relevant data. And as your colleagues already pointed out, we need to be thinking a little bit outside of the box. at sonars and 3D and other things that can augment these and make it better. Okay? Cool. So, please.
Don't you think over time though, the way that they programmed Watson to have an associative memory basically, they could just make that cheaper and easier and just upload it into every robot?
So every day we're making progress in terms of making computers faster, being able to hold more memory, and being able to solve a certain problem a little bit better, right? But again, we're talking about a huge leap here. We're talking about I'm able to model a problem very well like faces or street signs or kittens and then generalize into everything else. Think about how many things we got in our life. It's just insane. And so really what you would need is a picture of everything from every angle on every corner of the planet updated in real time all the time because things change.
Right.
And then also annotate it at all times, which is just insane. It's basically almost infinity, right? It just can't be done. Was there any— no. So let me just get back to this because this is very important. Like, what is the internet made of right now? Well, no, I mean, okay, I should—
okay.
What type of data? Text.
Text?
Perfect. Okay. Of course we have lots of text.
Graphics.
Videos.
More.
Videos, there we go. Videos, there we go. That's it. So most of the internet right now is actually made out of these 2 representations in terms of visual data at least, right? And they actually, they're dominating text and they're going to be dominating it for much longer. So we have these things called photos. What is a photo? Well, it's really just a slice of time and space, right? Or a moment in time, okay? And then we have video, which is just capturing time. You know, we've said that from the beginning. 30 frames per second, that's what we want to capture. Seconds, boom, that's a video. Why do we have these 2 representations? Who made them? What's happening here? Why is it like it's 2015— I want to say 2014— 2015, and then we have so many of these that we just can't imagine our lives without them? We're snapping all the time. Why? Well, so it's kind of very interesting. We have them right now because we've had them for a very, very long time. So it's almost part of our DNA these days. So 100 years ago or something, someone managed to capture light through a lens, made a film camera, and then computers came and we just digitized that. And so basically since then we haven't really done a lot of progress. Yes, more megapixels. Yes, better signal-to-noise ratio, et cetera. The fundamental technology is the same. We just, we have a deprecated format almost that we kind of brought over and it's basically kind of like took over and it's dominating the internet. Same thing with video, right? It came from Hollywood. They wanted to represent motion. Great. Again, 30 frames per second or 60 or whatever. We made cameras better, but it's the same fundamental thing. And here's a secret, and you probably know this already intuitively. We're just about to create an entire new internet in 2 years or 1.5, and then afterwards even faster, like a new internet and so on. This is a presentation that Mary Meeker from Kleiner Perkins gave About 2 years ago. And she was kind of predicting at that point based on the current sort of amount of visual data that we're creating and sharing on the internet, how will 2014 and 2015 look like? And this is how they look like, right? And again, it's just about to get bigger and bigger and bigger. So in some sense, nothing that we've done before matters anymore. Like all your pictures, they just don't matter. We're just about to take them all over again, okay? That's pretty cool, right? And you can think about it why, because we have so many devices, Phones and tablets and watches and glasses and all those things coming with cameras. And it doesn't matter that you've been to the Golden Gate Bridge 1,000 times, and it doesn't matter that the perfect picture of the Golden Gate Bridge has been taken by some photographer, right? So you shouldn't take one. It doesn't matter. You're going to go there. You're going to be like, hmm, it's kind of cool, foggy, whatever. Boom. Snap. Why? Because it's all about you at that moment, at that time. It's about capturing a feeling, an emotion. And then 20 years down the line, you might look at it or not and remember that day, right? Associative memory. So this is all real. So now the question is, if we're starting a new internet, should we be starting it and just kind of, again, like just copy over, replicate all these old technologies? Or can we think about something else, something outside of the box?
