Spatial Computing Catalyst #10: Capturing Humans for Synthetic Worlds with Just a Camera

So I'm one of the co founders of Radical, he's just joined the team. He was originally an investor but luckily he thought we were cool enough to also become operational. I'm the CEO but I co founded the company with an AI scientist and most of our team is actually almost all of them, with the exception of the two of us, are computer scientists. We're about 14, 15 people now and I always say I'm entirely inadequate for the job, but luckily nobody else wants to do it, so that's why I'm sitting here in front of you. In short, the idea was born out of a recognition that the 3D content pipeline is still awkward, slow and expensive. And I think if we can do, if we can provide a contribution towards making it faster, less expensive and quite frankly just more user friendly, then I think that might be business.

So you're doing the same thing for capturing somebody's body is like 6D AI is doing for grabbing them. So might be interesting if you guys end up working together. You know, they grab the couch, I've.

Heard a lot about them, that'd be awesome.

And then you're gonna capture the human gestures in there and train new things.

So explain where we are today and perhaps where we're going to be in the next year or two if we, if we do our job right. And one of the ideas that we have for today, meeting, if you guys agree, was to actually sort of not tell you everything that will tell you what our vision is. I suspect there are things that we're missing, there are things in the big picture that we're missing out on. Most importantly also, we know that we have a bunch of skills in a bunch of areas, but we are probably not very deep in any one of those. We're not experts in any one domain expertise. I am not personally. Yeah, the two non technical guys, I, I often feel, let's just see how public this is going to become. We all often feel that we need a lot of help.

If you become a billionaire, very public, you'll look back at it and go, man, were we dorky or what? I have a few of those videos, I'll take it.

So for example, Anand actually as we were talking about getting into this meeting, mentioned a great point and that is we've never thought deeply enough about what the user experience is going to be in ar. Yeah, we turn it on to make sure that people know that we can do it. I think probably like a bunch of people are doing. That's presumably why a bunch of people.

Are Implementing AR or an AR ready.

Or compatible type solution. We did it practically for the same reason. We just wanted the industry to know that we were AR ready. But I think the time has come where we are going to move. We want to drive more engagement. And one of the ideas that NAM had when I thought that was great was perhaps to provoke you into reaction. See, I think maybe, maybe you have an idea, maybe not, as to how this could be progressed.

Irene and I study the spatial computing world and there's a lot of companies we met with Arty that's making a virtual being. Right. Sort of here. But it's very incomplete, very early. It's good enough for a full phone for 20 seconds to capture a little video with yourself. And it, you know, they're building things for the movie industry so they built a spider man and other things that are coming out this summer.

It's called RD Arty. And what do they do?

Sorry, virtual. Virtual being or synthetic character. Right.

And. And how do they animate the character?

It's using some of the AR cloud kind of techniques of knowing a little bit about the. The world around you. Just scans here. Right. So I can see it standing here. Right.

But it's really doesn't animate the body from within.

It does, but it's not a script. It doesn't.

That's what. So maybe that could be a collaboration.

Yeah.

So we can move it around in real time.

Yeah.

At least that's the idea later this year. So I'm just going to show you what we have here. This is our desktop app. The desktop app is purely a visualizer. That means it's not the AI. It also does not allow you to record a video and upload it into the cloud where the AI currently runs. This is a beta we released in September of 2018. The moment we released it, we stopped developing it and started working on gen 3. This is gen 2. So I'm just going to try and make it somewhat obvious what we're doing. So what you see Here is a 2D video that a user has uploaded. Conventional 2D video recorded with a conventional consumer grade camera. In fact, this was procured from YouTube so we don't even know how it was recorded. Our AI detects and reconstructs the motion of that person in full 3D and outputs fully rigged skeletal data that you can then use to put any character on it.

Wow.

So these are just some of the photo realistic.

Started with a 2D video. You figured out where the body parts are for the AI or Whatever.

Exactly.

Figure out where the body parts are.

Exactly.

Very, very awesome.

Thank you very much. Like, there's no dots.

No, no.

This is a. This is real.

It's off of a YouTube video. This is not real life. It's real.

Now we can also show you like one of the.

So imagine. Congratulations. Think about what you guys are building and what. What's happening with these four. These 360 cameras. These are going to be on iPhone, iPhones, two, three.

Exactly right.

So I'm going to be able to grab you out of the air like this in a360 video or on glasses. And anything you do, I'm going to be able to build a virtual thing. Okay, so those are those like tracking.

Apps that if you move around your face like that, it'll rebuild like some sort of mesh like that.

But you can hold it straight now. Yeah. How much computer power did it take to do this is on an iPhone. This is.

This is gen 2. This was actually. This video was recorded on. On an iPhone. But the AI does not run local readvic yet. It still runs in the cloud on pretty. This one does. Gen 2 does. It runs on fairly demand. Very large Nvidia GPUs, Q100. So if you click highlight the menu.

So dance. All right, so I just interviewed a AI student at Stanford who is a former dancer who's using choreography of dance to teach robots. She would use your thing to do a dance or to think through a movement of a robot. Teach the AI how to do that.

I'm actually. I would have to think about this for just a moment, which way the learning would actually go. We might learn from her or she might learn from us. So maybe it goes both ways. So our AI is composed of. Of a bunch of disciplines. They're fairly standard convolutional training methods in there, where we are limited by the data that we put in there. But we also have a generative approach to what we do. Most importantly, in the next version of AI that you'll see out there, Gen 3, what we've actually done, we've built a latent space that embodies it, that represents human motion and full 3D.

Right.

And I emphasize the word you probably didn't pay attention to. Most people don't. It's motion. The latent space represents human motion in three dimensions. And then it can take very noisy, imperfect data signal, such as from a 2D camera, from an RGB and map that very noisy, very imperfect, very sparse data to that motion model in 3D and then generate this out of it or something much better than this, hopefully, because this is Gen 2, which is our public beta now. To answer your question. To answer your question, I hope I can. One of the ways in which we are training our motion model is to make the model understand that the human skeleton is effectively a control system. That's a key word in robotics. The human skeleton is nothing other than a control system to understand where motion comes from, how rotations work, what drives, what do you drive? Something for forward. Do you invert power and velocity and so on and so forth. So I suspect that a lot of the terminology we use to train our neural networks of internal abc, so to speak, might be somewhat familiar to her, if not largely identical. Right, so if she comes out of the robotic space, then she'll think about.

She'S working on autonomous cars, robots and ar.

Well, if she think it.

So, Dan, she says she's a freshman student at Stanford. She has six years ahead of her.

You're mentioning her and she's at Stanford.

So she's really brilliant and she's a former dancer. Right. So she understands how humans move and she's trying to teach AIs how to move.

I'm wondering what do you think some of the holes are that you're missing?

So I mean, one of the things you wanted to talk about was ar, right? So here, this is actually a girl that works here at Beta Works and this is her. This is I just, just filming her. And that video was reproduced like this was took three or four minutes to come back from the cloud. Gen 3 will be like 20 seconds that the latency is like completely vanished.

So how much work is done locally and how much work is done on.

The cloud at the moment? Absolutely everything. I think Gen 3 hopefully is just a few days and weeks away. But, but once Gen 3 is released, that's first released into the cloud.

And give me the basics of what it's doing. When it uploads a frame of video to the cloud, what is your system doing? It's trying to find what, what did you train it to do, to see in that video.

So it uses a combination of.

Off.

The shelf commoditized neural networks that develop that produce the sparse signal, which is a 2D signal. It looks at the skeletal information that's not invented by us. We don't claim that it is, it's highly commoditized. But what we do with that as far as information is our secret. That's what we pride ourselves in as our innovation. So the initial signal that we extract from the video, which is the very first step, is just skeletal joint information on a two dimensional matrix. So xyz, sorry, an X and Y axis matrix. And that's all we see initially. What we then developed is the AI behind it that takes that very sparse information and converts into 3D motion. Does that.

Yeah, I'm just trying. I just wanted to see.

How long did it take for you to build that?

About a year to get to this point.

Yeah.

And that was just a beta, just a public beta to hopefully show off to the world that we're capable of delivering something that is, that is absolutely unique. And we'd like to think that we're ahead of the curve now. Of course, it takes a lot to remain ahead of the curve. I think there's a lot of very, very talented people working in some elements of what we're doing. But we're the only ones I'd like to think that are focused on delivering a solution that is usable by the 3D content pipeline industry. Which means that we are actually developing this as a tool that allows you to extract the data that the eye AI produces in the form of skeletal. Skeletal rig. But that's a download that information. Okay, I don't want to test account right now, but that information is. Comes as a downloadable file that can be used as an fbx which we made. Okay, so a lot of people in the 3D industry actually know that term, others don't. But an FBX is immediately usable and Unity Autodesk, Unreal, and so on and so forth. So we want that data.

So a VR developer could use this technique. In a few minutes you'll get a file back to drag into Unity to start animating a virtual character or whatever.

That's exactly the right question and that's exactly where we're taking it right now. So this was developed knowing that we would have to just still work in post only. Right. So this 10 second video to three minutes to process.

Now a developer like that today probably has an Nvidia 1080 card under their desk, right? Can you do that locally or does it need to be.

That's a P100, a V100 right now, but the P100 V100 will deliver 10 seconds and about 15 seconds on day one that we delivered into the cloud. So it's about 20x improvement that we'll get out of gen three, we think. And this is potentially maybe the single most important thing I can say today is that we hope to be able to deliver this in real time on prem. Which means supported by relatively strong GPUs like this one. Although of course these are not Nvidia GPUs. I think once you get over to the Windows and PC side you'll see much stronger GPUs there. We think at 1080 should. Should deliver real time on prem within six months. Maybe within three months just on some testing that will be at 20 to 30 frames per second. We're aiming for 30 at a very minimum to really get you a good performance within three to six months.

How about also kind of thing.

My understanding is the GPUs are fantastic on all of those. But I.

Because that'll be A lot of people will be getting requests and wanting to capture like her dancing Exactly. In VR and then grabbing her and bringing her into environments like a Unity or whatnot.

I don't want to forget your question. I want to answer that very quickly. So what I want to answer. But there's a lot of.

We both have different. She comes at it from the finance.

Come out of the technical space. What we're really keynote. I'm getting better at is.

Making this.

A more enjoyable experience for users that we didn't expect would even be interested in this. When we launched this, we launched this for independent professionals. Computer graphics professionals. The 18 year old kid in a Wisconsin basement living with his mother all the way up to the single A's, AA's, you know, maybe not the Disney's and the Blizzards of the world, but certainly anyone in between who I actually think are going to drive the 3D economy in the future. It's not going to be the Disney circle. They can't produce enough content. I like that. It's really fascinating. I think we need all those kids that today use Adobe and so on and so forth for two years.

I'm not bureaucratic.

Sorry.

I'm not bureaucratic either. I'm not going to issue a public opinion on this. But my point is I do think that there are 25 million independent computer graphics enthusiasts professionals out there today.

Yes.

And I think that number will grow to 50 million or so within the next few years. And they need a much better pipeline. We can't solve all of it. I think other companies are perfectly positioned to do it like Unity at the top for us. But I think Unreal is doing a fantastic job. Obviously. I think Autodesk might go in the same direction as well at some point.

I also think Blender is coming.

Blender is doing. We really love Blender.

Advances in Blender have been insane. They just released a video to coincide with their 2.8 release of something produced completely within Blender. And it looks as good, if not better than things you see coming out.

Big shout out to spring. Yeah, spring.

There's a whole. I mean, that's why Marcel is here. There's a whole bunch of people who are creative and are trying to come up with new ways to communicate on Instagram and other places. Right. And soon we're going to have more new devices that are going to make us very hungry for this kind of.

I still want to answer your question. So we did not expect that as part of this launch we would get any interest outside professional or semi professional segment. And yet about 30, 40% of our users, it turns out, probably are not that they're probably using it just for fun. They're consumers all around the world, especially Asia, Latin America, the Middle east, countries like Pakistan, Saudi Arabia. Completely unexpected. And I think there's a trend towards self virtualization. I don't think we're giving them what they want. I want to be absolutely clear, I'm not saying that for them right now we're not fast enough. We don't give the gratification that people see. We don't give them enough to play with. This is just a prototype.

Yeah, it's a start.

But I think we have an opportunity here, especially with the AR side. We were hoping that perhaps you have a reaction to that. How can we gamify this to make this more entertainment?

A non technical direction.

Exactly.

A social direction.

Exactly.

A Giphy direction. It's upstairs. Right.

We're in the right spot to talk about that.

Right.

And what do you mean by that exactly?

I'm watching my 9 year old kid and he watches memes all night long and he's getting very advanced on that and he's starting to want to make his own. This is going to let us act out scenes, doing things and share that with other people and entertain each other. And then there's a whole bunch of other things like coming, let us play games.

It's making me think that in addition to human beings, you would then need.

To have a little bit of a different library of other objects.

Exactly, yeah.

This is just an empty space that we created. Right, yeah.

So even with the objects that you have, you'd be able to then change those objects in a way, dress them.

Up like you would. Or as my son says, dad, you need more inventory. We need more inventory of things to.

That's a Unity app.

Right.

So I Mean, what our, you know, is like, you can populate that with environments, different characters, animate different characters. Like you can have your own content studio on Radical. Never have to leave.

So that's the beauty of it, right? So we actually are very quite seriously thinking about building this on top of, say, Unity. And this is giving people access and maybe Sketchfab as well, maybe have a partnership with them if they're willing to do that. And to basically give people access to all these amazing models that have been built out there and to make them available for people to use and then also give them.

We were just at USC and met with Sue Gar, who's a student, they're just graduating. And he has an animation tool that doesn't work on the humans, but works on everything else. Physical things, right? So I could make this animate through space, right? Virtual thing, right? You have the human equation. You add it with his equation and you add it with sketchpad. Now you have something really interesting, right? A new kind of creativity environment that lets people create.

That's awesome.

All sorts of stuff. This kind of work. You know, you go to Coachella. Have you ever been to Coachella?

I have not.

You're sitting in the Sahara tent. There's a hundred LED screens literally all over, right? All around. 10,000 people in this thing now. 30,000 people dancing, right? The entertainment that's up on those screens is going to be created by things like this.

So we're also going to wear. Are going to build the amazing artists in the space. One of my. One of my favorites, and he's a rather well known rockstar in the Unity space, is a Japanese digital designer and he's much more than a designer, so I should be careful about this. I'm not doing him. Just artist Kijiro.

Yeah, yeah, Kijiro.

And we're so luckily we've already. He's given us permission to use his special effects in our app. So that means you're going to get these insane particle based swarm effects where you just. The characters half dissolve as they move around and the visualizations are responsive to velocity and energy. So it's sort of physics based.

Marcel, are you getting excited over there? We're talking Marcel in his language. You should be like.

Join us. He was our man here at Peter Woods.

The guy brought us in.

No, this is really cool stuff. I can see a whole bunch of business model questions you're going to have, which I know Irene is.

I didn't properly answer your question, but I.

Well, my question's really big so.

So a big funnel at the top.

Yeah. There's a lot of directions you could go in.

So this.

That's what we're. Your question to her was a 68 page report that will come back to you telling you what is the likelihood you're going to have a business in this new space that she's. We're all ripping apart the markets because you're building new things and you need new ways to explain it to investors and to understand it yourself. Which path should you go? Because you're going to have 20 paths to go down.

This is actually one of the shout out to beta works. I'm not saying that because the distance. No, they actually keeping a very open mind on that front. We came in really wanting to solve a problem for professional or semi professional, what we call the individual contributors. Yeah, right. Independent professionals. Independent content creators is actually really sort of what we settled on because I think that captures the sentiment of most of these people. Their game development, that's what they call themselves. Film, tv.

Yeah.

Commercials, creative agencies and the app developers that I think covers sort of the. The largest segments of our target audience. At least that's what we always felt we wanted to do. Now getting that right from a business perspective, not just so much is it the right tool for them and so on, but delivering them exactly what they need at the right time, the right kind of pricing model as a fine balance. There is a case to be made that we just built the right thing for the right people, for millions and millions of people, and just make it a highly engaging platform, maybe with more social aspects, give them access to more characters, make it less about the workflow. Right now we have a strong bias toward the former, just to be clear, because we come out of that space, they come out of the enterprise space. And I just see the most hunger in the 3D content industry.

I mean we, we just saw evolution. We just saw 11 million people watch Marshmallowing Fortnite. Right. And he was getting a performance live. But there was lots of other things that were going on in that space that could have been captured by a dancer or a performer or, or us. All of us, right. We all want to do interesting performances. That's why you go to Coachella, because we all want to dance. It's not just the DG on the stage.

So I've had a vision for a bunch of apps that I could see myself, see myself using. I think many others once we have the real time home application. So this runs on your desktop or laptop sort of Consumer grade gaming computer will do the trick. What are they around a thousand bucks these days for? You know, we'll be able to deliver on prem real time version that then you take the contents of what you have and you stream it into a live virtual environment where people can collaborate.

Yeah.

The very first version I had in mind because people like to do dancing. Like I don't know what it is, but almost all the consumers that get engaged here, they dance.

Yeah.

We have teenagers in Pakistan dancing on our app. Right. And so the idea will be that they can stream their virtual representation into a stream space where they actually are social.

Right. And there's a dance floor by the end of 2021. We're wearing glasses that have 8K screens in front of us. Virtualized screens and sensors that are going to sense four cameras probably. Magic Leap has four cameras. HoloLens has four cameras that can see everything as we. Don't I want to create in one of those tools. Don't I want to watch things in. I mean, he was getting.

The thing that I just always want to highlight. And that's something very important to understand. We reconstruct the world in full 3D, even though we see very little. What I can actually do going forward with our technology is I film you from the front, but I can give you a camera view of your own virtual representation that can look at you 360. 360 while you would do it.

Yeah.

So I can be dancing on the table here while I'm moving around and.

You can tell the camera, you can show me myself from all angles even.

Though it's a single camera.

Yeah.

Wow.

So you can, you can for the first time experience your own journey through some world, whether it's a virtual world or somebody, probably a virtual world of some sort, maybe a mixed reality.

And you experience.

You can look at yourself.

We're capturing this with a 360 camera. Right. Why can't this live stream to your servers? Why can't it capture all of our emotion in real time and put a virtualization of you in the cloud in my glasses? Why are you. In three years, why are you not. Why aren't you doing this remotely? Why aren't you sitting there remotely in my glasses? Because I'm sitting in front of a 3D camera and you're sitting with a little camera like this. And you're sitting in front of a little camera like this. And your software is growing.

I have a vision for your product that's a little further down the line. Do we just give me one minute.

I can give you as much time as you want. I just want to make sure that.

There'S two other teams.

Yeah, I know. Sorry. We're just.

So basically, I see this as a product.

Okay.

Consumers can use it, but I'm not thinking of it as a consumer product right now. I'm seeing it as a development tool by Hollywood studios who will be. Once the visuals become more refined and more true to life, which is going.

To take a few more years.

You could actually have an actor or actress that you have in a database. You have another real person who's doing everything and then you can have that masked so that the actor or actress is now the person in the movie or the visual. And I know companies that are working on something similar, not to that full extent, but that's the vision for the future. So that they don't. The actor, actress doesn't have to do the acting anymore. They're just using her visuals or her. His visuals. And to have the entertainment sell.

Right.

So the vision of the future we got from Artie, who we met.

They know it's going to take about five years before there's some kind of like really true to life movement and look and feel to the artificial character or the synthetic character.

I do have response, but I'm not sure how much fun I'm giving. I don't think it's five years for us. Yeah, I think it's about a year for us.

Well, that would be great.

So we.

It's very useful.

The thing I want. I just wanted to put in there very quickly our motion model, which is the latent space that contains the information to represent virtual human motion.

Yeah.

Only contains beautiful, plausible human motion. It does not know anything else.

Got it.

This is.

Or something.

Yeah. Okay.

Good afternoon. To get all the names.

Where. Where do people. By the way, where do people learn more about you guys? What's your public.

Getrad co. Getrad co.

Okay.

We. We've heard that we ought to change that.

And we will get co. Yeah.

Oh, they call Radical and the.

The.

The app is called motion by Radical Radical.