Radu Rusu, CEO of Fyusion, explains how smartphones can capture 3D scenes in real time using a lightweight format called "fuse." The technology lets you move a virtual camera freely within a recorded space, turning phones into 3D capture devices for everything from indoor mapping to autonomous vehicles.
Chapters
Can anyone tell me what's missing? You already said 3D, but I guess more specific. I'll show you. This is what's missing, right? Are you Star Trek fans? Anyone? A few. Okay, good. A few. Good. So what's missing is what we call space capture. We have time capture. We have slices of space and time. We don't have this yet. Wouldn't it be cool to have this, right? I can just move around freely in space. bullet time or whatever these things are called these days, right? Slow motion. It doesn't really matter. This is what we would like to get. So to revisit again, we got something called time and we got something called space, right? Obviously, time never— doesn't stay still. That's just like one of the universal truths of all, you know? So we're always going to be capturing some time when we capture space, okay? And we got 2D photos right in the middle. And then we got things like video. And animated GIFs, which are kind of the same thing, right? And then panoramas going in that direction. And so the place where I'm coming from, this company called Fusion, has been working really hard in trying to go into the space capture modality, as you'll see in a second. So from our perspective, we feel that this is something that hasn't been really tackled. People have been talking about 3D for a long time. But concretely, how do you bring that to consumers? Millions of people use it without even thinking that it's like a technology breakthrough or something. It's just kind of cool and useful to use. So this is what we call a fuse in our own language. And it's really just a format for space capture. And it's really made for what we call Gen 3, which is like your smartphone, and then useful for Gen 4 wearables, and hopefully useful for robotics when they come. So you mentioned that you're working Okay? So those of you that know anything about 3D, you've probably been seeing things like these for a while. These are— this is what we call a point cloud. It's a collection of 3-dimensional Cartesian coordinates points in space, and they're presenting a scene. And you can acquire these 3D point clouds using a variety of devices. Everybody heard of the Kinect by now? It's a camera that can actually give you depth images that afterwards you can convert into something like this if you want to. In this example, you see kind of like a line laser. So imagine a robot arm or someone basically doing this manually too, moving a line laser up in space and then measuring distances to a bunch of different points and then creating this representation. And most of the world that's in technology, when they think about 3D, they think about something like this. Now go out there and talk to consumers. They don't care about this stuff. All they care about is— A phone. The Matrix. Can I move around in a scene while stuff's happening, but move really slowly? Can I have the ability to control the camera position? That's pretty much what consumers really want, and that's exactly what we discovered, too. So you know this thing already, hopefully, right? For the first time ever, I believe, we just surpassed usage on mobile versus desktop. It just makes sense. All of you are— yeah, you have some laptops, but most of you have a mobile phone right now in your pockets. Okay? So Gen 3 became more important than Gen 2. And again, I'm just going to be talking to you a little bit about the kind of things that we're trying to do at Fusion. We're trying to sort of address this end of the spectrum. Forget about PCs. They're done. We've done a lot of those over the last years. So for those that don't know much about us, we're a very young company. And this is not a pitch. It's just telling you some of our background, where we're coming from. We have been building technology like this for, me personally, for 10 years in various institutes, but it wasn't until recent where we saw this opportunity to bring them to consumers because again smartphones got to be powerful enough that they could run these type of algorithms in real time and cameras, pardon my French, started sucking less. No more VGA cameras, now we have beautiful 60 frames per second, half HD or full HD, So you can actually leverage that thing and use them for something meaningful. So in terms of the company, we've been around for about 2 years now, I guess. And we closed a Series A. Not important necessarily. And one of the things that we've done before this and for a long time is this initiative called the Point Cloud Library. So any of you that's interested in 3D processing and want to understand more about what we think is the future, you should check that out. It's a full open source project. Right now it's the largest open source initiative on the planet in 3D. It's basically backed up by a ton of— like hundreds of companies and universities and so on, very large community. And we created it out of the Willow Garage because the purpose that we had there was that if we're ever going to have robotics happening in our lifetimes, we need to go open. You can't be closed. You can't just do it by yourself. This is so difficult, as I explained earlier, right, that you have to go full-blown open and have as many people as possible on the planet contribute. So again, some of the applications that we've been working on just to show the kind of things that are possible these days using software like PCL, but also I guess a lot of other types of libraries and research available online. You can do lots of interesting things like create maps. That's actually the top left part of the figure with the Kinect. You can go around a room. And you have this process called registration where individual views are kind of like placed together. And then you're going to end up with a beautiful CAD-like model that might be useful for something. I don't know what, but it's useful for something for sure. Obviously, you know about autonomous cars and/or semi-autonomous and the way they're sort of like looking at building, incorporating these technologies into your next vehicle that you're going to purchase. So here you see an example in the middle Where all you're doing is you're taking this 3D data and you're sort of trying to figure out what's planner and what's not, because obviously planner is really good for wheels. So that's something you want to make green and the car can just go there by itself. If something pops up from the scene, like if you have an obstacle, it's going to be marked red and you shouldn't drive. The car should stop. And lots of other things, again, in terms of recognition. In the bottom part, you see, again, the PR2. This was way before— like, I think this is 2008 or something, or '09, where we programmed the robot to actually go search for doors using that line laser sensor. You see the sensor kind of going up and down, and then figuring out where handles are, how to go and open them, and move through, right? Like, go through the door. Again, by itself it's not impressive, maybe, but think about this in the context of a bigger application where this is a— capability that you need to have and many other things. And as I said before, you should definitely check this out. This is a pitch. Like, now I am pitching you to go and check out open source software out there because it's really wonderful and it's going to help you a lot if you ever want to get into this. If you don't, then don't check it. And it's about 3D point clouds, not cloud computing, just so we're clear.
