How 3D Photos will Change the World | Fyusion CEO and Founder, Radu Rusu

So how did we look at this problem? Well, we really wanted to have something that captures space, as I said, and not time. So that means that if I move around an object, it shouldn't matter if I move in half an hour or 10 seconds. The end result should be the same. I should have sort of the most compact, minimal representation of that space. And then I should be able to navigate in it. And this is a major difference between this and a video, right? If I move around something in half an hour, I'm going to get half an hour of video. And then 10 seconds, 10 seconds of video, right? So really I want to have space and not time. So again, same thing I mentioned already to you. You have photography and video dominating. And you have this thing that goes in the direction of sort of like space called a panorama. Panoramas are great, but just so you know, they're also 2-dimensional. And usually the way we represent them is as a projection on a sphere or a cylinder. And one of the problems that they have is that it can only represent the world outside. So you're usually in a bubble in the center of a sphere. You're taking a panorama, and that's the stuff that we can represent. We project it on a sphere. It's great. You can't go around an object. I can't move around my laptop and represent it as a panorama. It just mathematically, it's not possible. The other problem with panoramas is that they can only represent static scenes. So if you've been moving your hands while someone is taking a panorama, you're going to see you appear with like 5 hands. Okay? That's not cool. It's not cool. And it turns out that people actually don't care if statistically only like 1% or 5% of that panorama is broken. They're just gonna reject it. They're gonna think the whole thing is broken, 'cause who wants to see someone with 2 heads? Not cool. So again, you try to move from this to something a little bit different, and this is again the format that we came up with called Fuse. So Or for the purpose of the patent, it's called a surround view. So what this is again, first of all, it's trying to leverage existing technologies where we have this wonderful sort of beautiful touchscreens and gyroscopic sensors. So I want to get that and make it part of the format from the beginning.

Is yours really made to be on mobile, or can you zoom in and it still has the same resolution?

You can zoom in. And then you can also embed it in websites, of course, because as I said, although PCs are kind of dying, we still want to be able to embrace that. I mean, we want to have them be useful there. But yes, you can zoom in there.

Is it vector-based?

It's not vector-based yet, no. As you'll see in a second, in the next slide, it has vector information behind the hood, but you still have to render it somehow, and there's some limitations there. I can talk to you about technology in detail if you're interested. So it's not vector-based. You're still rendering on screen something like an image, right? Because that's what people are used to. In our studies, actually, we experimented with a lot of different outputs here, and we were showing at some point full 3D meshes, and people are like, oh, this is kind of cool, right? I got like a full 3D mesh. I can 3D print it. Great. That's fantastic. But with 3D meshes, you can't represent motion, so things have to be static, and also every now and then you'll get some artifacts, as you'll see again in the next slide, And that— people don't like that. People don't like weird stuff like triangles popping up in space and whatnot, right? So as panoramas, 3D meshes are not consumer-friendly so much.

And with this technology, you lose audio?

There's no audio. We don't care about audio. We have— yeah, this is space capture. In fact, it's going to be really interesting if someone figures out how to integrate audio with that. I want to hire them for sure, but we should talk about it.

That's what we're trying to do.

Do you think that point cloud library stuff that you're doing, somebody could leverage that to add audio in?

Or just to connect, maybe not super related to this.

So again, everything that we've done there is visual. So audio is very interesting because audio is very one-dimensional. Audio can only represent time, right? So it's kind of interesting, how would you move— how would you actually experience audio in space?

But we do, and if you— your ears are able to perceive the different directions.

That's true, but it's still time-based.

It's how the frequency travels over time at a certain speed.

Yeah, exactly, because if I stop audio, If I pause it, I get nothing. I get silence. You see what I mean? If I stop a visual representation, I get something.

I don't have the facts to back it up, but I'm thinking that there's a way to make that more—

add another recursion. Let's talk about it. That's very interesting. There's a lot of—

sorry, go ahead.

How was the talk?

I'll show that in a second, the next slide.

Thank you.

So across the street, there's this thing called Twinlin, which is a 3D scanning photo booth. Do you have this application used for that kind of stuff?

Yes, absolutely. Is yours better than theirs? Sorry?

Is yours better than theirs?

I don't know. I haven't seen theirs.

They have a really intensive setup.

It's very different. So just to explain a little bit this process of 3D scanning, so we've been doing this for quite some time now in different sort of examples where we get people in booths and so on, and usually the output there that people are interested in is a point cloud. It's what I showed you earlier, right? Because that's useful for 3D printing and it's useful for a bunch of other things. However, one— Look, look, look, it's this guy.

Oh, he's using it.

Oh, oh, oh, oh, I see, I see, I see. I completely saw it. What's happening? Something on the screen? The problem with 3D printing, guys, and I'm sorry to burst your bubble, but if you're believers in 3D printing, it's never going to be as big as 2D printing. Because we've been printing by manually writing on walls, on caves for like thousands of years. I don't even know how much, right? And so 3D printing is a little bit limited from that perspective. I do think that it has limited sort of applications, but I don't believe that it's gonna be as big. So that means that from a consumer perspective, a lot of people are like, okay, this is cool. I'm gonna 3D print my whatever, this, and then that's it. 'Cause how many objects can you really 3D print in your house? So launching an application for 3D printing like this might or might not be a good idea. Our purpose here was to not output, like, again, formats for 3D printing, keep them under the hood, and I'll just switch here so you actually understand how it works, and then give back people a way to navigate in that space. So it's all about visuals at this point, right? Again, we have movies, we have photos, how do we move towards something else that captures space? And this is just the first kind of tip of the iceberg. the first thing. I'm sorry, I think there was a question there and I—

I was just curious if I understand it correctly, what you're doing is kind of inverse rendering and then putting the sort of texture that you captured visually back onto the 3D model.

It's a bit more complicated than that, but the process here is just to answer another question is, so you go in space with your camera and your camera unfortunately captures time, right? I mean, there's no way for me to tell my camera, give me data at like 10,000 frames per second. You just can't do that. There's a shutter. You guys know the principle of optics and all that stuff. So I'm going to capture time at some frequency. In this case, for example, for iPhones, I think we can get 8 megapixels at like 10 frames per second or 15 frames per second. The latest iPhones can do that. Full resolution, some frame rate. You capture all that, and then what you're trying to do, you're trying to reconstruct Let me just play this again. Something that looks like this under the hood. And we called it— we defined it already as a point cloud. It's a collection of points that geometrically represents the scene. However, this is just not interesting for consumers. It's useful for you. If you're a geek like me, you like that. That's cool, right? I'm going to look at that. I've been looking at that for 10 years. It's amazing. I don't think my mom wants to look at that, or my wife. She doesn't care about this stuff. How do you go from there? You have data in between. You build this graph. How do you go from there to something where you can have this smooth, infinite interpolation data that, again, represents space? So that's where the challenge is.

Were you at the San Francisco Makerspace last year?

No.

No?

OK.

There was someone there who was doing something similar to this.

Yeah, so lots of people are trying this. And I think that's really wonderful because this, in our opinion, is the future for the reasons why I explained.

This is a lot more in-depth. I thought maybe it was you and you pivoted and so—

Could you ever do anything where instead of moving about it in a circle, I could just take my phone and do this and then it could almost like using the circle shape to create the rest of the shape?

Yeah, yeah, yeah. So at core, the technology can be allowed to do many things like that. The challenge, and I guess you're all entrepreneurs here, is how you package it up and you create something that's extremely extremely simple. It's almost like it's dumbed down in some sense so that you get massive adoption. That's the key. As with any other experience that we have right now, like many successful companies that you've heard of, they create something very simple at the beginning, and then they keep on adding more. If you create something too complex, it turns out people don't like that. It's just like, ah, man, I've got to move my phone way too much. Just give me something simple.

Boom.

So the way it works, as I explained, you just move your phone in space. We're trying to simplify the problems I explained, and you have to move in a certain direction. So don't be doing like this because we're not going to allow it. And then you're just— the first basic thing is you're just creating this band, uniform band, smooth interpolation of space. That's all you're trying to do. And you can go in any direction. I mean, you can go up. You can go right, left. It doesn't matter. But this is the first step, first milestone, getting something like this and making it really cool so that people enjoy it.

Okay?

So just a few things, a few kind of lessons learned from the kind of things that we've been doing. As you might know, Apple doesn't have a beta— they don't allow beta apps on their App Store. So we actually had a very simplistic release of this technology last year. And you have to call it 1.0 or something, otherwise they won't accept it. And then what we did is we gathered feedback, lots of feedback. So over the course of some months, We just looked at what everybody was doing, talking to them, understanding, is this something that we should be even doing remotely? Maybe it's just not interesting. Maybe people are like, you know what, it's cool for one— it's cool the first time I try it and then I don't need it anymore, right? So we learned all those things and then in December, right before Christmas, we launched, I guess, our first experience on iOS and then Android as well with the kind of things that you see here, right? So really nice simplified UI. Also bear in mind that We're trying to make these useful for people that are interested in a social context. So we learned that by mimicking an experience they already know, like Instagram in this case, that helps a lot. Because people are used to doing these kind of things. You don't want to bring too many different things in the same application. If you already sort of built a lot of technology and you bring that in by the space capture through the format, you don't want to change the UI completely and then change some other things. That's going to throw people off. So keep something stable. Experiment with one. See if it works. Boom. Then try another one and so on. Don't try to change too many things at the same time. Obviously, because we're doing something different, now you have to have guides. People need to be told how to move their devices and the kind of things they will get. If you just leave it— the first beta version or alpha version that we had was not showing you anything. It's like, good luck. Just point on the— point at something and move your phone, and of course the results weren't that great. Once you tell people how to use something, things become much easier, right? One of the things that we also realized is that a lot of people like categorization. We've been talking about Mechanical Turk a little bit, but that's something that you get paid for. But in general, like just people enjoy to sort of collect things, right? So we do it in the space of photography, but I guess recently with social media, we sort of like didn't do it as much because Instagram just blew up and everybody has a feed of some hundreds of entries, and that's all they do. So we looked a little bit at other companies. Pinterest in this case actually was almost an example for us, the way they've been managing to get a very large community of people curating things and clustering them together. So we went with the same approach. We need to have galleries. People just love to pull together their own, but also other people's representations into these galleries. There's many interesting things like You want to have channels with notifications. I can actually explain this to you later, how it works. Messaging, right? Wow, you want to have messaging in the same application? Yes, because you want to have one-to-one direct conversations with people and show those people fuses. Like I'm at a store, I look at some shoes, I'm going to go like this. It takes like 5 seconds. Send it to my wife. Should I buy these? It's much better than a picture, obviously, right? So I don't want to post it on a social network. And then get likes on it and then erase it. I just want to have a direct conversation. So yes, there's functionality for messaging in there that works really well and people are very happy with that. Okay? And here's when you start adding some layers of technology on top, right? Because obviously you can't just leave it at like that. You need to be providing a little bit more. So there's something that we call visual tagging. And I'll show you some examples. The way this works is— so now that you have 3D information under the hood, you have those point clouds and whatnot, right? You can basically almost add physical tags in the world that become sticky on those objects, and as you rotate a scene to see something from different sides, they rotate with the scene as well. So it's pretty cool, right? And I mean, there's something that we should have had this for many, many years, but again, it just wasn't possible algorithmic-wise and so on. So now you can have your hashtags there as— One of you already mentioned that, or user tags or URLs. Like, I hate the fact that so many social networks out there are trying to prevent you, keep you like sort of grounded, and they're not letting you like sort of get out of their domain. If I like a pair of shoes, I should be able to say I bought them from there and let people go there. Like, what's the big deal? Why do I have to keep everything constrained, right? Because they're going to be using this more afterwards because now they can tell people directly And then you also create like a nice sort of like ecosystem there, mechanisms for people to get referrals and click-throughs and so on. Okay?

Go ahead. Are you like a— because I saw you had it there for like model with clothing. Are you afraid that because your app is free that like now any clothing designer can see that and put these pictures in?

We love— we would love everyone to do that. That's our goal. So as I'll explain towards the end, our goal is to create a new internet. Right? So a new internet cannot be created by a single application and like we control everything. So we want people to use this application and embed these things on their websites, and we'll release an SDK so they can embed it in their own applications and so on. We just want this space capture to become the next big thing, because if it does, we might be able to build robots on top later. Who knows?

So you're creating a trend so that the world follows something that's going to provide us with useful data? Yes.

Anyone, like just—

You said earlier that you want to integrate with PCs, so I was wondering, like, how are you going to rotate?

Yeah, so obviously, you know, we can't take our monitors and do this. That would be ridiculous. So what we've done is we created viewers for Flash, HTML5, and applications, dedicated applications for Facebook and whatnot, where you go and you drag with your mouse to move it around in the scene.