Spatial Computing Catalyst #5: The Movie/TV Studio of the Future at Intel's Volumetric Dome

Sa.

So what are we looking at here?

So we're in a 25,000 square foot sound stage down in Manhattan Beach. It's Intel Studios. To my left here is our capture dome. It's 10,000 square feet. We've got it set up today with 105k cameras, very large installation of SkyPanel from Arri. Here we do performance capture, creating essentially videogrammetry and volumetric video for the entertainment and sports industry.

Very cool. And who are you?

I'm Dave Smitty and I'm the head of product development for Intel Studios. Very cool.

Thank you.

So in classic computing terms, garbage in, garbage out. So you have to start at the top of the funnel where the data comes in to. That's the key point to start. And then there's steps all along the way through basically to the final point where the photons are coming out of a screen or coming directly into someone's eyes. So that whole pipeline has places, points along the way where you can affect fidelity or affect quality or affect accuracy or however you want to define what that is to make the experience the best you want to be or the kind of experience you want. So I look at a company like Scatter, who's using with very small cost or limited resources, producing really high quality content. It may not be high fidelity capture or high fidelity volumetric, but it's really interesting artistically. We did a piece with Reggie Watts, captured 14 dancers in here and showed it at Sundance in South by Southwest. And it's not photorealistic camera capture, but there's really interesting work you can do as an artist working with point cloud data and shaders and so forth development tools that we've created to make something that's artistically compelling. And we've talked with a number of companies out there and there's this sort of. Is photorealism the destination that we're all trying to get to?

That's a question I have.

Yeah, yeah. Or is it really creating art or entertainment or content that's compelling to an end user? And there's great content that's low fidelity. Even the 2D world, that has an emotional resonance for people. And I think we're really trying to service folks that want to create emotional resonance or create great stories or create great art and understand that there are traits and characteristics to the format of volumetric that you can work with as a creative tool. So if you're a traditional photographer, you can create low res black and white photography that has a lot of grain into it. You can play with Depth of field or bokeh and create something that's very interesting and compelling and emotional. But maybe it's not high resolution or photorealistic that you might see with someone who's out shooting with a 4K camera trying to get the highest resolution possible. So I think it really depends on what you're trying to achieve as a creator. Are we in a place and time now where you want to shoot something with a beautiful actor or beautiful actress and then have it on a big screen close up shot of them crying? Probably not. That's probably not what you want to do. That's going to be very tough to achieve. But there is a path technologically that we can get there that's related to software and related to hardware and that's an evolution path that we've been on for decades now.

So I'm going to let Robert talk, but I have one other question. So you're talking about art. So there's an artist called Chuck Close.

Yes.

Who does the photorealistic. Basically it looks like a photo. You go close up and it's actually not a photo. He does have a way of making it look like art even though it looks like a photo. So I'm wondering how far into the future do you think it'll be before we actually have photorealistic volumetric?

Well, 10 years. 10 is such a nice number to say. So everyone's like 10 sounds good. I think we're 10 years away, probably realistically. And I think we're on a path of. We're in a place of experimentation now and understanding what is the experience. Where do you stand as the viewer? What's your perspective? Is there a proscenium or not playing with all those things in the interim while we develop this technology to get to a point where like, well, I really just want to play with it and mess with it and I want to play with bokeh and I want to play with depth of field and I want to play with perspective and let the technology develop and as a point. And I also want it to be high resolution and if it's high res you can still have all those choices. So in the interim we're playing with the levers that are available to us. Yeah.

It'S interesting being here. I mean we're here in front of one of the largest volumetric movies, TV media, sound stages. What do you call this place?

Well, this is a soundstage and this is a capture dome essentially. Right. So but I mean it's a place place for ideally it's a place where storytellers and creatives want to come to. To play with this new media. Yeah.

Irina and I are writing a book called the Seven Visions, which is seven industries that are gonna be disrupted by spatial computing technologies. And entertainment is certainly one of them. And this is sort of like ground zero for where the disruption is gonna happen. You know, cutting edge. Right.

Big dome here. So I can imagine movie producers coming here and wanting to have something up there and then people below here. So you actually have a set that you capture the whole scene, which you can't do in a smaller scale, kind of green, you know, in a dome.

Right.

So this is very special. And you're saying that you think it's.

Probably the largest one as far as we know. We don't haven't seen or heard space.

It has to be be done. Intel obviously knows what they're doing. So maybe if somebody tried to rig up something like this, they wouldn't know exactly technically how to do it correctly and have the storage and all that stuff that we saw.

Right, right. So there's a significant amount of compute here. The amount of data we store is quite a bit. And I think in the future we'll be seeing generating sort of two to three terabytes per second coming off of the capture system. So we're going to be quickly, by the end of the year, we're doing shoots and we shoot an hour here and we'll fill up a petabyte of data.

Where I was going was by 2030 on the way up here, walking up to this dome. I asked you what percentage of movies are using green screen techniques? And you said it was like 85% already. That shows a huge shift in how movies are made over the last decade or so.

Yeah, it's a tremendous amount of computer graphics, CGI animation work that's done. And there's a lot of animation films being made as well that mix animation with regular humans.

And so if this thing is successful, and I bet it will be, how is the jobs going to change in this industry? What kinds of jobs are going to be more in demand? What kind of jobs are going to be lessened to man? For instance, I know or I've heard that a lot of the physical props have gone away. Right. That you're doing everything digitally now. So now a guy who used to manage that inventory of physical props doesn't have his job or her job anymore.

Right.

But there's a whole bunch of new jobs coming because of volumetric studios like this.

Exactly. So I think We've seen the growing numbers of 3D artists and modelers and animators. Those jobs are growing and we've seen decreasing jobs and some moving the sort of physical atoms around. Right. So we're sort of going shifting from atoms to bits in a way. So I don't know what the percentages of like the atoms jobs are going down by X percent a year and what's going up on the other side for the bits jobs. But we do still see art props people. We still in the digital world. In Greenskin, you're still putting out set pieces there that they're interacting with, but all the backgrounds of the environment will be CGI and they'll integrate seamlessly with the physical props that are out there. So wardrobe is still quite important in a lot of jobs, a lot of movies, but the wardrobe is becoming more sort of cg. I think the other thing we're seeing.

I met one of the guys who does costumes for Star Trek on the way back from Dubai and he says, oh yeah, my job is to put little white dots on people. His dad did the costumes for Ben Hur. He said, my dad actually had to do real work. I just have to put white dots on.

That's right. Basically putting mo cap suits on people and then putting markers on faces and so forth. We've also seen a tremendous explosion in this area with the TV production. Sort of the. The golden age of TV is back. Right. In a lot of ways. And those productions on many levels are as high quality as you would have seen in cinema in the last 20 years or 30 years. But there still is a lot of work that's moving atoms around in the TV space. I think the areas where it gets really interesting is where you start to integrate artificial intelligence in with how we create the end assets and end materials.

Yeah. Like magically Paz Mica that they're showing off.

Right.

A virtual being, polygon based. Instead of physical bits like you are, it's a polygon based being that's going to talk to us and interact with us and hand us things in movies.

And stuff like that. And being assistant and someone who's there to sort of make our life better. Right. I think that's the. The best vision we have.

Or just entertain us. Yeah. Fable Labs is building Lucy. Right. I mean, there's a whole bunch of people who are trying to crack this art. Arty is doing different things.

And so we've experimented with AI here for things like style transfer. We have digital assets that we've created. There's point Cloud data. And we can style that with different techniques. So shaders, for example. So we'll take as input various images or kinds of artistic materials and use a neural network that has different human in the middle levers that we can adjust to apply that style. So we're thinking of how do we get artists to be the human in the loop, the artist in the loop with artificial intelligence and the captured assets that we create here to sort of have the character and the scene represent what's going on with their internal world and have that representation determined by the sort of partnership between the machine and the artist. And that's doing research on that inside of the company. In concert with what we're doing here, we do capture for what we want. Part of the goal of what we do here is we want to represent the human performance. And so we're interested in not just people having dialogue around a table as part of a narrative experience, but also people moving through space. So dance, performing arts, sports, and capturing that and then representing that in a 3D way that looks like it's authentic. It actually happened. It wasn't built up from the ground with animators, which can take a lot of time. It can be very expensive. So we're using that sort of capture and we're combining that with AI. And this is the long term vision of what's potential is that it looks like it was performed by a human, but it was procedurally animated by an AI.

The body motions.

Yeah, the body motion. So you could drop. We could do a capture here of someone running around. Let's say maybe it's a chase scene, but they just run around in like three or four different ways. And we use AI to generate them running around through and dropping an ar and they could run through the streets of Los Angeles or San Francisco, New York and they would be able to buy the glasses, can understand the environment and understand the conclusions and the different services that you have to run into. And they would avoid it. Yeah. And that would be done by. It would look like this is going.

To be a nuts future I'm at. You know, another friend is building virtual cities somewhere in China. They don't exist.

Right.

They're virtual cities and they can pop them up. Another company I visited is building autonomous cities.

Right.

For autonomous cars.

Right, right.

Completely like you ask it, you just talk to it and say, I want Mumbai downtown, I want it to snow, I want heavy pedestrian traffic. Right. And it just builds it live based on what you want.

Right? Yep.

And soon we're going to be Walking around with actors captured in here.

Right. So we could, you know, you could. You could take. Capture those actors here and get there and do like some skeleton or pose estimation. And then you could actually change their wardrobe and what they look like. But it looks like the actual person moving as opposed to one that was generated from the computer by animators.

Isn't that crazy?

So you know what it makes me think? I'm thinking now this is much further down the line, right?

Yeah.

40, 50 years from now.

Sure.

After you have your database of objects and people and faces and yes, there'll be new actors and actresses, but you can create them digitally. Right. To be fidelity, to have fidelity to realistic ways of the way the face should be in the body should be.

Right.

So I'm thinking somewhere down the line, you're not even going to need physical actors anymore. You're not going to need physical sets. You can compile it all together in a virtual world and it'll look extremely real.

Yep.

And in that case, then Intel Studios would be something else. You know, you'd find another business because.

You still need computer.

It's going to take a lot of time and a lot of other companies and other things happening before that becomes the case. But I think that's where the future is going.

I think you're right. I think we're all trying to get to this place where the outside world truly responds to what's going on with our internal world. And you can express your own stories based upon on how you're feeling on the inside. And all of the materials that we have, the sort of 3D materials, can come into life in that way. Right. I think that's the ultimate vision. And that certainly, as you pointed out, is going to be generated by massive amounts of compute that's always available and delivered across a network at a very, very high data rate. So it seems real. Right. So we're in this sort of sub, 20, sub 10 millisecond way to get. Get that data to us. It's very far away.

One of my dreams is to be able to blow up a building.

Right.

Live. Right.

We'll actually be able to do that because all the buildings will be digital anyway.

I know, but to destroy a building like this digitally with a bomb, it takes. How many hours of compute does it take? Right.

Yeah.

When you watch Hollywood blows up buildings, all the. What's that?

Fight Club didn't look real and they.

Blew up the building.

I know, but I'm thinking like, you know, a standard 20, 19 movies today where you blow up a building and you have beams that are deforming and stuff like that, right.

It takes a lot of people in their tweaking things. So it's not automated. Right. It needs a lot of artistry. I think in the future the artistry will be built into the app.

Yeah.

And then of course, somebody will oversee what the.

But even the modeling of blowing up that wall, right. And having it bend properly and deformed properly takes a lot of math, right. And it takes a lot of compute and takes hours, right? It takes hours to blow up and.

It sort of become. But it's. If it's something that can be procedurally rendered based upon your request, right. That you may pull from. You know, there's 30,000 different versions of buildings blowing up or walls blowing up that can be selected for you. And perhaps based on your feedback, there's some kind of back into the system, causes it to generate a new version of that. Your point about that? There's someone overseeing this. I still think there's the human in the loop, understanding what's the intent? The intent and also being able to express what they want to see as the artist that's sort of bringing these elements, these visual experiences, auditorial experiences, sensory experiences to people. And I think even today we have this world of like, you can be anything on the Internet. You can do whatever you want on the Internet, you can see whatever you want on the Internet. There's all kinds of potential for interaction. And even most people, it seems at this point in life want either a lean back experience, like, I want a really great story told to me. And then there's some who are like very. Who are very active browsers, active searchers, actively involved. And there's like the 8020 rule, right? That's your principle. And I think we'll see that. Like, oh, I want to see, you know, who's. Who's gonna be the great creator of the future in 50 years, who creates this world, right. I want to go to that experience, right. What's the Disney version of that in 50 years?

Right.

What's the, you know, like, what is Lady Gaga create a world for people in the future? The world. Lady Gaga is 50 years. Right. Those kinds of things. And I want to go see that. Or I want to create my own. And then there'd be different tools for that.

You know, the kids today are playing with Minecraft and Roblox, and my son's playing with tiny town in VR and building an entire land. Right. We tweeted it out. So People could see what. But that tool is not close to what you guys play with here, right? You know, yeah, you can't blow up buildings, you can't animate people, you can't animate anything in that kind of tool. Right.

Different scale.

But I think the creators, the creators are going to come out of those feel, you know, out of playing with 3D, Minecraft and Roblox. And they're growing up in a world where they expect then to lay on top of it. The technology that you guys are developing here, it's pretty interesting. I mean, Irina and I dream about a world where we are wearing a pair of glasses that shows us, you know, some K screen, you know, 4, 8, 16K screen in front of us, maybe more someday, you know, but we don't yet even have 1k screens in front of us that are really good. That world is coming. You act a little humble and say, we don't know what's going to come here. But you know, we know Apple's coming in 24 months. We know intel is funding all sorts of different projects. We know that Magic Leap is coming with a pair of glasses next year and on and on. What does it mean for Hollywood when people are actually wearing glasses that have a pretty good monitor in front of them so that they can play with augmented and virtual reality stuff in their homes? And what does it mean for this kind of studio?

But you know, I think Hollywood is about telling stories. I think that's the fundamental goal of the system here, enabling artists and creators to tell their stories in immersive ways. I think the cinema has been immersive for a long time. Television can be immersive as well. And Hollywood's played a big role, I think, in the world of virtual reality, both through all of the insights and knowledge that come from the gaming space, but also from the insights and knowledge that comes from the narrative storytelling space. And Hollywood is on all platforms where there's an audience and desires to consume content. Hollywood would like to put that content there, there. What it means for us is, I think for us, our current target really is to deliver performances that can be experienced in an interactive fashion and in a real time 3D fashion. And for me personally, I'm very interested in how we can mix reality with the virtual reality. And that's something where we create these sort of real time performances, we reconstruct it, we and put it into another context. And for me, that creates, I think, really interesting opportunities for storytelling about mixing media and remixing that's been going on Remixing media has been going along as long as media's been around. Right. Like drawing on a cave or doing collage or working with assemblages or doing installation art. This is something now that we're taking these 3D assets and putting places that may not have bend and then being able to modify that environment other ways. So changing the pictures on the wall, the floor, blowing up a wall, whatever that is, that's to me is about remixing. And I think there'll be really interesting stories told with that. That kind of genre.

Yeah, I think a lot of it, what you're talking about is personalization. So get to the point where you create your own story.

Yeah, I love that.

Yeah. I mean, a lot of it is written already made or it could be done on. On the spot in. But it would in a brilliant way, not in a badly animated way. And once you get that, you have people that. There will be master storytellers that didn't even realize that they could be master storytellers. Now they have the capability of showing that.

Right.

Or people read a little script on their phone, do it, do a little bit, and all of a sudden they're in a movie. Right.

It's much easier to.

And Greenberg talks about that all the time, right?

Yeah, absolutely. I mean, because the cost and the technology and you know, all that kind of stuff, for many years, many, many years, people are used to ingesting that from the screen.

Right.

And being passive. But once they can become active doing that, I just. The imagination is like amazing.

Yeah, yeah. I love the personalization angle. I think that's been a big trend for computing as well over the last few decades. Is getting down to sort of a one to one relationship with the user, giving them control and creative tools as well. And that's self expression that we've seen in the last 15 years online. I think that's a really good point to think about. I know that Hollywood's looking at ways to bring the viewer into a more active participant as well. And we saw that with recent productions where you sort of choose your own adventure. Right.

Bandersnatch.

Yeah, Bandersnatch. Right. So I think that's a really good case in point. And it's such a diversion from the lean back, tell me the story and I'll just follow along different paths. I think there's a lot of.

I was just flashed, my mind was flashing some of the magic that I saw at Stanford's VR lab. You know, Jeremy Bellinson showed me the power of embodiment and then power of. The power of being immersed and the empathy that you can come from being inside the head of a character. Right. Or a person or, you know, and Chris Milk made me a bird, and I was instantly flying. Right. The power of embodiment is something that Hollywood's going to play with for decades, I think. Right.

Yeah. And there's two paths we'll see this go down. I think at least one is the power of embodiment and sort of becoming part of the story. And then there's this. How do we also create realistic experiences that enable people to have objectivity at the same time so that they can experience something with some amount of objectivity? I think there are lots of stories that there's some benefit to that. This kind of cracks that open, and I think forces ask a lot of questions. The lights have arrived. And for me, personally, I think there are different kinds of stories where I want to be. I want to be in their head and experience that, and I want to understand what I'm missing by being part of that, their experience and seeing from their perspective. And there's others, like, I'd really like to be objective. This is a little too scary for me.

Frankenstein.

Yeah. I want to watch it happening, right? Yeah.

You don't want to be the person who's actually in that experience. You want to kind of understand what it is, but you don't want to be that person.

Yeah. I think there's a spectrum to work along with that objectivity to sort of being subjective. And I think these technologies are kind of. Kind of mixing that up and causing us to think about what's the right way to go and how to play with that. And I think there's benefits, really strong benefits to both. And the ones I think, like the VR experiences that were horror experience, some of those are really frightening and difficult for people. And I think you need to enable some layer of objectivity and distance from those kinds of experiences.

Ah, My son wants to go straight in.

He's like, I'm in.

Yeah. I was watching an LBE where people were running through VR and there was a really scary scene in there, and one of the women just threw off the headset and just couldn't handle it. And, you know, other people found it very enjoyable to be scared like that. Right, right.

And you can choose that. Like, do I want to see the latest horror film? And, like, if, you know, I don't like horror films, I'm not gonna go. I love. It'll go, but it'd be nice to have some kind of settings like you can move back and forth, like, oh, I really understand what this character's going through, through and embody them. Right. I really want to understand what it's like to be, to be this dinosaur that's running across the plains versus, like, I just like to lean back and kind of watch.

Yeah, it's like it, it's not. It's like a 360 plus version where you're inside the 3D and you're the spectator and you're watching everything happen around you and that's. That actually nothing exists like that right now. That's really great vision, actually.

Yeah, yeah.

Irina and I are, you know, doing this book and I, I'm just keep flashing on the tech industry is about to give human superpowers and Hollywood is gonna make me a super. Have superpowers.

Right.

I'm gonna be Spider man in the movie of the future. Right. I'm not gonna have to watch Spider man for all around New York. I'm actually going to be able to do that. Yeah, yeah. Is that crazy?

I mean, and if you could actually haptic, like, if you feel it, that.

Would be even, even, even further. Right?

This is all like, that's a ten year thing.

Sure.

Plus, well, in 40 years from now, you're just going to beam that thought right into my head. Right. But for the next 10 years, you know, I think Hollywood studios like this are gonna make me like, be able to be a dancer. Right. Or be able to get inside, you know, a famous actor's head as he's saying the lines right. Or she's saying the lines right.

And I think the, when I think about it, sort of, there's a lot of dialogue you have about sort of Silicon Valley in Hollywood. And like the storytelling component I think is really important. The visual artistry. That's really, it's really strong, deep DNA here in this town. And the enablement of sort of amplifying that is the strength, I think, of Silicon Valley. How can we get broader, distribute? How we can make that happen faster? How can we make it happen more real? These are technologies that I think are delivered in the best way partnership happens.

How can we make those chips smaller?

A future where I have a digital double that can be rendered out in 3D and also can perform tasks for me, right. It's my AI helper, right. My real assistant is the creation of that, all the algorithm and the compute processing with the Silicon Valley effort, I think the Hollywood effort is how do we take the sort of Visual and storytelling components and add that into my digital double that represents me. So the things I might say when I'm not answering my phone. My phone rang earlier. My digital double, answer the phone, speak in my voice. What emotions and tonality would they use with the person who called based on the time of day? So all the mixture of those two have to come together. So I think there's a real kind of.

If your wife starts screaming at your digital devil, the devil goes, hey, we better get the real guy on the phone.

Yeah. I don't know what to do. I don't know what's going on in that other reality. I have a feeling of what's going on with reality at least. But I think that's. Yeah. The world is changing very rapidly. Right. And I think that the partnership that's happening between these two domains is really compelling.

It's finally coming together. Because I know that for a very long time, Hollywood and Silicon Valley were kind of thinking like they were at opposite domains.

Yeah.

But this is the. Is so different now. And I love the way that it's coming together.

That's why Irene and I got together. She's Hollywood. And I'm so nerdy. Although you're nerdy Hollywood, for a Hollywood person, you're pretty nerdy.

See what I mean?

Yeah. Yes. We have this thing going. Right. That's cool. Yeah.

But you're right. You're right. The combination of the two, the storyteller and the person building the chips and building the. The computer vision and building the AI, the math part of it underneath is needed. Right?

Oh, yeah.

You need that. The aspect to update and automate things and save time and money and all that kind of stuff and leave. All the creativity comes in a different. In a different way then that other stuff which takes up all that time, which isn't necessarily creative. It was just like a person was needed to actually do all that stuff. You can't do that anymore.

Yeah. Today, like Intel CPU drives something like 75% of render for Hollywood.

Yeah.

So. And when you. So you have. And so there's always constant requests and how can you change your architecture? How can. What's the roadmap? How can we produce movies not necessarily faster or not necessarily cheaper? All that. I'm sure those are requirements. The ones that are interesting. Like you have animators that work on virtual consoles come in and they're in the flow and they've got to have a render happen very quickly as they draw in and change a little part of the frame and renders out and they get the feedback. So how can that loop happen? Very quickly and if there's slowdowns and they're in the flow, that they want the creative spark happening as much as possible. It will be always available when the creative spark is in place. And if it's not, then they just timeless animators. So that's just the power of compute. Making a creation of this is like.

Gang and Emmy award. The music's coming up. That means we gotta end. Our mic's getting cut off. Well, thank you so much for giving us a tour around. What do you call this space again?

This is the Intel Studios stage in Manhattan Beach, California.

Very cool. Thank you so much.

Thank you. It.