Spatial Computing Catalyst #15: Hollywood Goes Light Field

So we're playing around with cameras today.

Yeah, we're here. We're here. We're playing around with cameras.

I've been telling my readers you're the Ansel Adams of the current age.

Oh, that's a great honor. Ansel Adams is one of my icons and heroes. And thanks for welcoming us into your world. And this is our world. We're radiant images. We work in camera technology to break the status quo of how you users are able to engage with each other.

And you just. Your firm did the Beyonce music video. You do all sorts of.

We do all the crazy. We do all fun, crazy stuff. We like to do really interesting projects. So some of the things that we've been experimenting with is what if we started looking into light field and volumetric and what we call free viewpoint videos, where the audience is no longer bounded by 16 by 9 screen.

Screen.

They can immerse themselves inside the actual movie. Yeah. And they can interact with the actors.

So. So we're starting to play with new cameras.

Yeah. Right.

This 360 camera.

Yeah.

If somebody wears a VR headset and watches this video somewhere, that could be right here.

That could be the third person next to us. Yeah. Hi, people. That's everybody. And then this way, it's completely opposite instead of us outward facing. What if we did the same concept and we took a bunch of cameras and started looking at you now, right. From multiple different vantage points. What that would do in a couple different methods. One, let's just call it volumetric. That allows us to capture the subject matter from so many different camera points that it builds a point cloud. And that point cloud gets. Sometimes they apply a mesh to that. And what that means, the bottom line means that your video file has depth information into it. So if you use the new phones or new headsets or computers, you're able to move inside a subject.

So volumetric means I capture you as a 3D thing. I get a point, point, cloud, or I get a mesh of little triangles.

So you get all the details of depth. And then we texturize it with my texture back on. And then now the audience can walk around me at any point and see the video from any vantage point. We're no longer bounded by a 16x9 that we can't get inside. Data object is that I could be teletransported into someone's living room so I could be brought into someone's house. So imagine the power that's going to.

Give us Princess Leah's little hologram in Star Wars. Right. She Appeared on top.

Exactly. I use that reference all the time. How Princess Leia was able to be brought in. And there's no geography. She, she was some police house. And her telepresence was viewed by Luke Skywalker. Yeah, Luke Skywalker.

So light field. This camera does volumetric and light field. Right. There's a little difference in light field. Right. Explain what the difference is.

So we use the sensors, cameras, and we try to make them smart by putting lots of over sampling by data. And that data allows us to see things that sometimes the human eye can't see. So let's just look at it from one vertical. Let's say if we were interested in using it for inspection and automation. Right. If a piece of art comes in and you can't tell the difference between the two because they look like the same color and look appearance wise, identical. But there are hidden treasures. And every time someone uses a paint stroke, those are depth information, Those are the DNAs inside the painting. So what we're doing is we're capturing the artwork, we're peeling away the actual paint. Now you actually see my fingerprint, which is whatever the artist drew. So now we can apply it for intelligence. The other thing we can apply it for is entertainment. How we've been told stories and how we have envisioned ourselves. What it would be like to be in someone else's shoe, to be in somebody else's world. Now it could be more realized. Now you could physically feel that world. You could be part of that. Let's use Godfather as an example. We could be inside the movie. Yeah. And you could interact between Marlon Brando and Al Pacino. And now you're the third person. Or if it's a boxing match, you're now the referee watching the boxing match from that vantage point.

And Jeremy Bailenson at Stanford taught me that one of the magic you get with VR is embodiment. So maybe I become one of the boxers.

Well, how crazy would that be?

And see it from like if, if we were capturing the scene on your camera, I could jump into your eyes and then jump into mine and then jump back in the box. Right?

Absolutely right. The exciting part about this is.

Go to God, viewers.

Exactly. So the exciting part about this is there's been a lot of what ifs in technology. What if this happened? It's no longer that question anymore. It's no longer like, what if? It's when do you want it? When do you want it to. Now. Now. Right.

So we gotta wait till 20, 21 and enjoy this on our Apple Glasses, right?

Yes.

Or Apple Visor. They're not gonna call them glasses.

So the great thing is that what we try to work on is technology agnostic. We're creating the form, not just us. There's a lot of amazing companies that are fully focused on using multiple cameras to drive depth information. So the audiences no longer have to watch things on a flat screen. They're inside the movies, and the movies to come inside their home. So that's where it becomes really interesting. That technology exists currently. Right now, if you take your simple Amazon app, the one that's built in, there's an AR component. You can bring furniture inside your own house and you can see what that furniture would look like in your living room. Right. Why can't we start bringing actors in your Beyonce, Michael Tyson, my friend Cassie is a Beyonce. Or better yet, one of them being Rob Brunette.

Right. Beyonce is better than Meyonce Fan.

So. So that's part of what we do at Radiant. And then we're also experimenting a lot with artificial intelligence. How do we start training the AI so that it could follow, capture, track, identify, better enable technology to move forward? There's a lot of exciting things that we're interested in that could apply to autonomous cars, factory lines and so forth.

Cameras like this one. I mean, this is only a little half pipe.

Yes. This is again, prototype two. This won't capture.

I imagine that sometime someday soon you're going to have a camera like this in a football stadium or a basketball stadium, right?

Absolutely.

To capture the playing field in volumetric and in light field.

Some. Some light. The good thing is that that experiment is already in motion. Yeah. If you go to Intel's stages. We did.

We went to their studio and they.

Had a big dome. Pretty big. Pretty big.

200Ft across.

200Ft across. So you could fit half a court. You can fit a half a court, but you can do an entire basketball easily inside there. Right. So that's the method that intel is moving forward to. We're more interested in another metrics, which is how do we make things portable, deployable, how do we make it scale? How do we give it to thousands of people to create content by using cameras that are affordable. Because when I say 100 cameras, we're not Intel. Intel's great. They position themselves to be the leaders in that. But we're not Intel. We don't have hundreds of millions of dollars.

They had a data center underneath the dome.

They don't have a data center. So we were all about deploying this whole thing.

Is run by one laptop, by the way.

Exactly. So we trigger the camera, synchronize them. Synchronize to multiple devices. Not just cameras, lights that are synced perfectly to the cameras. A lot of devices. The main thing is that we're all about deploying it out there so we could scale. Not just have a huge stage, that's deployment and content drivers have to go inside of it.

What is a frame like in one of these cameras? Is it. What would you call a frame? Because when I shot a frame on a Kodak camera or a Nikon, you know, you had a good idea of.

You took.

You press the button, you get one frame.

Yeah.

What's.

So these are hundreds of frames. Hundreds of frames that when you join it together and when you finish your solve. Currently, in Volumetric, we export OBJ files. In Lightfield, it's still proprietary. There's different camps. There's no. Here's your file base currently. Yeah. That is being developed by lots of very smart people that are trying to figure out a standard for that so that it can get exported directly into Unity. I know all toys do some amazing, amazing stuff in there. And I know Google's moving forward on that as well. The most promising thing I saw was Faunhofer, which has a plugin built directly in Nuke that could work into a visual effects pipeline. So when I saw that, that was like, oh, yes. Because tools have to be as easy as a remote control, it cannot be an obstacle for the content creators. We have to be like water. We have to flow, make it easy so that people can experiment. Because it is early stages, we experiment, we learn things. We learn things that burn us, things that work, things that don't. And that requires removing the technology barriers and putting up.

This is a new world. I mean, even with the 360 camera. This camera custom.

Back in 2015.

No. And we were talking before I started the camera on that, that we need a new language for this because we used to talk about close shots, wide shots. Well, now this camera does them.

All.

Right.

And you choose.

And the viewer chooses.

Oh, you choose. Well, I can.

I can put the camera up high.

Right.

And now everybody can see.

Exactly. They can see the entire thing.

And I can put it down by your.

And by the way, you notice if.

I spin the camera, it.

It automatically.

It automatically keeps the horizon. Right.

So that's exciting. So you could do that or we could reframe for 16 by 9. So we work with a lot of amazing experimental filmmakers. Early days where they took over, sampled it and then they reframed so they could tell their story from any perspective. So there's some really interesting things that can happen. And also light field, volumetric. It doesn't mean that all your content is for that method. That's what we recommend. We recommend that our audiences taste and viewing mechanisms have changed the way they engage, the way they see things. They don't watch it, they don't go home at 9 o' clock and watch CNN or Fox. They're. They're seeing things when they want, on demand, streaming, on mobile devices, on their news feed, so forth and so on. So we have to be content creators, have to be adaptive towards that. So we have to expand.

Now I know how a camera like this works because I got a look at Lytro's prototype at Stanford University and it's actually capturing all the rays of light and the angle that light is hitting your face.

Exactly.

Which means with this database that you, you get after capturing something in here, you could change lighting.

Yes.

Later on in the computer and you could do a lot of things, all sorts of, you know, take skin out, put different skin in, put different clothes on. People do all sorts of fun stuff.

Right.

And that's why Hollywood is so interested in a camera like this, because it gives them a lot of freedom creatively later in, in editing.

So the first major breakthrough is it gets rid of the green screen. That's one. And that's what everyone's aim is. Everyone's aim, including light field. When they first looked at the target, they like, great, we're going to remove the green screen. We want to shoot in practical locations. Imagine the revolution that's going to give us. We can now go into a Starbucks, capture Starbucks. We digitally own that asset. That becomes your IP now. Now we bring the actor that's not available to come to Los Angeles. They might be in Rome. There's a light field stage. They get inside there, they do their performance and there's another actor four months later in la. We could bring them in that photogrammetry scan and then we can change our mind. Say, you know what? Now for the Chinese market, we're going to have a different scenery. And now we want to move the camera anywhere we want and we want to relight it for nighttime. We want to change it. So that's the beauty. We consider this method. It's almost like a time capsule. It's a true capture of a person's asset that could be reinterpolated in all the different viewing methods as well as storytelling.

So how do you light an actor in one of these camera systems?

Good question. If it's volumetric, we light it very fast, flat. Because we want to get rid of any shadows. We want to minimize the shadows. Shadows create occlusion issues and we want to minimize those. When we're doing light field, we actually light like a cinematographer does. We light aggressive, we backlight it, we put harsh lighting. We want to make it edgy because we're recapturing. When you see me, you're not seeing my shape, you're seeing my light reflection being right. So that's what it's the cameras you're recapturing. So the more interesting you make it, the more interesting. And that's why at Radiant, we didn't focus on volumetric only. We feel in the ant. And a lot of the stages we built for some of the major studios are both capable. We have installed our stages on some of the world's largest movie studios, and they're creating both volumetric and light field. And this is where AI comes in. Because our vision is that you can do both. Yeah, there's a possibility that it could be both. And you're looking at a portal. This is our test and it was successful. And we no longer have to shoot light fields flat. Flat. We could curve around it and eventually capture the entire thing. And it could get very exciting.

How many frames a second are all these cameras shooting at?

So the industry standard, everyone, everyone from Microsoft, they're hovered at 30 frames. Sometimes they go to 60 frames. We feel for the actual capture, we want to break that mold as well. We want to move it to a lot higher frame rate. We're proposing and we're testing currently every single day, 500 frames a second, 5K camera. I know that's going to blow some brain cells. They're going to be like, wow, wow. But it's really important because if you look at everything that's been captured in volumetric and light field, they shoot very skinny shutter motion. They don't want any motion blur. So they want it. But the problem is that creates a little jaggediness, right? Yeah. The more frames we have, the more fluid it's gonna feel. We don't see the world at 30 frames. We don't hardly have 60.

We start to see it analog. And our eye can follow and move and predict even where my hand's gonna move.

Exactly. So that's how the human eye works.

Right.

We predict the next frame. So we're over sampling it for that purpose, but we're not shooting slow Motion. We're using it for the best hdr, and the best example of that is your iPhone. When you take a picture, what does it do? In hdr, it takes a whole bunch of three high speed images, stacks those photos together, and the algorithm gives you an HDR. Yeah. Imagine now 500 frames a second. I'm delivering 60 frames.

Oh, are you playing with the extra exposure in those frames?

Yes. So that's why lights are synced.

Oh, we haven't even started talking about lights. Yeah, but just the camera. By shooting 500 frames, maybe one's a little overexposed, one's a little underexposed. And that's why you could capture light field, because you're capturing all the different pieces that are going.

Our theory is that you can apply this also to volumetric, which means now we have that your HDR just increased. Imagine if you captured me with a traditional machine vision camera, which is what all the studios have right now. They have basically machine vision cameras. They have about 8 to 10 stop dynamic range. The color space is limited. They're not designed for motion picture. We look at the camera, we're like, it's got to be better. So high frame rate cameras are the perfect answer for that.

Now let's talk about lights. You don't have any of the strobe lights.

No, we don't. So we're still.

They're still secret.

But the concept I can share because we're so this.

These cameras are firing 500 times a second.

Yeah, not these. But our other cameras are too high frame rate. They're very small. They're synced to the light. So think about a strobe light. When the photographer takes a picture, it sends an exposure of light for that single frame. Yeah. And it doesn't waste any. It doesn't put a harsh light on something.

Now a flash is a millionth of a second. A shutter. If you look at film photography, the shutter would be open for 30, the 30th of a second, whatever it is.

And that light is synced to that exact shutter.

Right, Right.

So now we're doing it at extremely high frame rate. Crazy. A thousand frames a second. 2000 frames a second. Second. So 500 is easy. And we're pixel synced. Pixel, not frame synced.

And what does that let you do by shooting like a strobe like this?

Really? I didn't see it because our eyes.

Yeah, these are flickering at 60 hertz, 50,000 times a second.

So imagine if we went to 40th, you would see flicker and that's why old fluorescence, you start seeing the flicker. Right. So we're shooting at six such high frame rate that you don't see it. But it's constantly sending the exposures high and low, high and low. But the eye doesn't see it because the eye only sees 60. As a matter of fact, the only time I can show a demonstration of this is I go to 48th frame. You're like, oh my God, look at that strobing. And then we go to 60. You can kind of feel it. We go to 90. You don't see it. You know, I typically. 90.

Yeah. How many, how many flashes would be in a camera?

Like doesn't need that many. Just about. Good. Even lighting. If we're doing full volumetric for light field, whatever the cinematographer wants. Right.

So you might have one, you might have three.

Yeah. We'll probably do more like cross lighting. Yeah. If I'm doing volumetric, we're probably going to have a lot more diffused light from all the vantage point. All the areas of issues like underneath, like my neck is going to have shadows, I want to get rid of those. I'm going to shoot from here. The good thing is it doesn't disturb the actor. That was our first concern. Like, you're going to put someone inside one of our spheres, they're going to get blinded by the light.

Now just being here is costing us a few thousand dollars. Right. Using a camera like this is pretty expensive per second. I keep hearing terms.

No. So that's what we're trying to. That's what we're trying to break. We're trying to. Our mission is to make it scalable. I want to make sure that content creators have access, not excluded. And how we do that is by working the opposite way. Lytra did an amazing thing and we're all standing on their shoulders right now. Everyone that is doing multi camera volumetric light field, we're learning a lot from what Dave paved ahead for us. So great things that they've done.

Yeah.

We just went the opposite direction. And when we built our first light field rig for Sony for last year for nab, when we launched it, it was basically window panels and it was on a Pandora. Yeah, you have the.

It used to be flat, like.

Yeah, we just took all those cameras out of there. But we started with consumer grade cameras. These are $600 cameras you can buy at Best Buy. And the reason we did that is that was our entire model. We're not interested in doing one off projects. We want to have projects and capabilities for our content creators that they could go out on the field without computers and servers and all the things that get in the way of the free spirit of the Enzo Adams that to go on top of Yosemite, capture those. Imagine if you had to bring the entire lab with him, he wouldn't be able to do this.

Now, this is still a little hard to get to Yosemite.

Well, this is a pre prototype so that we can know that this method works. Then we're manufacturer. We have a system which is based on an auxiliary. Let's show you.

Here.

We position in here? Yeah, go inside of it. So this one is based on hexagons and octagons. And we position in this particular one. 10 Alexa Minis. We've done red, so we're cam, but any cameras and the OXA could be different shapes. So these are outward facing. Yeah, this would be in the middle.

Of the foot football stadium or something. Yeah. So in front of a concert performer or something like that.

Well, we built this for one of the largest, oldest studios in Hollywood. Can't tell you unfortunately, I can't say it yet, but it'll be publicly announced soon. We built this in five meters. Much bigger. All out of rigid carbon fiber. Super lightweight. The reason we selected carbon fiber, it's the best material for thermal expansion. So. So heat and cold, it's not going to matter. And it's also the best for energy observation. So we have to constantly recalibrate. Our patent is based on the fact that all of our structures are all curved and we know in space every camera perspective. So we could simulate a lot of simulations. And it gives us the most adaptive, lightweight, most portable volumetric and light field stage.

And you have a mechanical engineer on staff, right?

Oh, yeah, absolutely. So who. Who does? All of our amazing workers. Kurt Lin, who's our. Who's our materials engineer. He starts as a material, but he is a genius and he's done some of the most incredible designs. So he built our first oxa and Kurt and I met each other three, four years ago and. And we're good friends. Yeah.

Awesome. Well, it's an.

It's honor AI stuff too.

Oh, sure, why not?

Let's go.

Yeah.

So Omar is going to be our demo. If you want to go over there, Robert, to the monitor. Yeah. As you can see, it's recognizing Michael Mansouri on the screen. Yeah, right now. Yep.

He got me too.

Yeah, you got you there. It says Robert. It says his name.

See, I'll never have privacy again.

Now if we look over here, it says Omar, his age. If he's smiling, he's not smiling. Smile, Omar. He's smiling his eyes. Close your eyes. Omar, open. And then here he goes. This must be wrong because there's no way you're 50 years old. I'm 54. 45.

And the AIs are probabilities as you get more training data.

Yeah. So we haven't trained. We just this as a proof of concept for a major government industry and. But we have infrared built into this and everything is being processed in camera right now.

And look at that. My, my camera is really hot.

Which is, which is actually you can see inside the camera. So what he's going to show you next is object tracking. Object tracking. So it says it's a person want to grab the.

You can track the camera.

Watch this. Now he's going to bring a laptop in. Yeah, it's going to say there's a laptop. Yeah, See, it says laptop. Laptop. So's the laptop. Now if we put this on a remote head, it would track the laptop. If we said follow the laptop, the head would automatically track the laptop.

And that's the trick for that kind of camera in the future because you can have waves of those cameras identify.

Sunglasses, scissors, all the other stuff. Maybe this one's not trained yet. I don't think this one's trained yet. Yeah, but we could start training things one by one and then train it to also autonomously follow. So imagine what impact that will have for so you'll.

I've been talking with my friends that I have the most expensive camera in the neighborhood because I have a Tesla.

Yeah, you do.

And my Tesla's gonna someday autonomously drive. Right, so and it captures video already. Right?

That's exactly our point. Vision by itself is worth no value. Vision with intelligence, which is what Tesla's doing, which is the aim with Microsoft and also what Google are doing, it's incredibly important. It's the true essence of big data. Utilizing vision to make it smarter. Instead of using laser to give you information. Tesla believes we use vision to give intelligence. Because as your Tesla is to trying driving, it's just big data. It's gathering data for Tesla to make Tesla's AI better and better and better.

I'm donating the data from my Tesla.

Soon that Tesla will self drive completely on its own.