Spatial Computing Catalyst #1: Niantic's AR Leader: Getting the AR Cloud Ready for Harry Potter

I'm Robert Scoville and this is Irena Cronin. And we're starting today a new spatial agency called Infinite Retina. And Infinite Retina is aimed at bringing Fortune 500 and other entrepreneurs into the spatial computing world. And today we're at niantec and we're meeting Ross Finman, who is running augmented reality research at niantec, which is the company underneath Pokemon Go and soon Harry Potter. These are games that are played by more people than anything else in the spatial industry. So it's real important to start our video show with Niantic today.

And this is only the first of many to come.

I'm Ross Finman, I was previously CEO and co founder of Escher Reality. Now I help one of the partners of the AR leadership for Niantic here, particularly helping like architects in the future strategy aspects of the company.

And when I last talked to you, I don't know, 15, 16 months ago, you were running Azure Reality and we were talking about how you were building or visual building a slam map, which we should talk about. Is that still sort of where you're going with this fun company behind you that builds Pokemons and soon to come Harry Potter kind of games?

Yeah. So the term is now called, since we last spoke, the AR cloud. So then building out the kind of map of the world that everyone can interface with. And then Niantic is actually the former Google Maps team. So then Google acquired Keyhole, which the CEO of Keyhole, which became Google was Earth is also the CEO of Niantic. So that mapping is at a core part of Niantic. And then going after this AR cloud is a very big high priority for us.

Yeah. And how do you define AR cloud for a normal person? And I'll give a try and then see if you. It's a way to see the world with a computer and it puts a virtual layer on top of that world where you can put things on top of signs or games moving things. Right?

Yeah. So like just to start off with, you need to understand reality in order to augment it. So then the reality understanding part is both the what is like what is a table but then also how do you come back to these experiences? So then very similar to how you phrased it. It's kind of what I view as this mapping layer on top of the world, a t digital copy of the real world that you can then interface and build off of. So the analogy I like to use is it's like an operating system, so you have a consistent application layer on top of an ever changing world and then people can build off of that. Well, before you had Windows on top of different hardware underneath. Now you have a world operating system on top of an ever changing world.

Irene and I are starting a company called Infinite, which is going to focus on the business aspects underneath the companies that are coming into the spatial computing world. Do you have a question to kick that off, Irina? Because we can talk about the nerdy.

Part of what.

I'm starting this company with you is because that's what we really care about and that's our expertise and a lot of other people don't. So we have to know the technical aspects. My biggest question for you has to do with your vision for AR and what are those features that are out there that have to do with the AR cloud, those interactions with the real world. I know I can't ask you about Harry Potter specifically, but in terms of your vision of what you'd like to see and what you'd like to build for the future that way.

So. Well, starting out, since we're all nerds here, there are two main aspects that are different about augmented reality and how I see those building into the AR cloud. So then for ar, first the position of the device matters, so then you can move the phone around and that is really great for location based gaming. But then also we at Escher made a demo where you can play Pong with it. Yeah. So second is the real world becomes content and that's where I view like the vision kind of gets fulfilled and how you can begin to interact with the earth as if it's naturally there. So then like the way that animals and people interact with the world, like digital content should interact with it. So but that happens like one once you have this kind of real world understanding. So like what is a table, what is a chair? Like kind of the different objects around this room, but then also you have larger scale experiences. So one of the things that I'm most excited about, like the features of the future is in Pokemon Go, we showed video, promotional video saying, oh like 10,000 people rushing times Square to catch a mewtwo. And then they're all in the same world together and battling about like that. Having 10,000 people all together in one experience, it's kind of like a pop up stadium in a way to have people experience these massive things together. And that really touches at the core of Niantic is a really, really hard but very interesting technical problem. And one of the things that I'm personally most excited about.

Let Me just have a follow up for that. So you're creating a new reality.

So you have the augmenting the current reality.

I'd say that there, there's a reality that there is. And then with the game and causing people to come into one space like that, thousands of people, you have created that happening. All right, so it's not just augmenting, it's now creating new experiences that are built off of existing, the existing world in a way that I'm looking at it. Do you foresee that for the future of AR that there's going to be like a fluidity, a possibility for branch narratives that happens that isn't so?

Oh yeah. Like, I actually think in the future we're going to have different channels that you can subscribe. So then if you want to have a Halo themed day or a jungle themed day, then you can have your own narrative of how you want that day to go and what it all to look like. So then the buildings can have vines growing on them, going around and you can be in this world that's still part of the real world. So it's a little bit like the Netflix series Stranger Things where it's just the same world, just in a different dimension. This gets a little less like concrete and what that look like. But I always love the concept of subscribe to your reality. Like, what is the world that you want to see and how do you want to see it? And then what is the like personalized experience? Because like if, if I love cats and you love dogs, I should see cats all around in my world and you should see dogs all around in your world. And like that, like just the number of times people can like smile and enjoy all that, I think is like really, really awesome. Now backing out, like, what are all the different tech parts that you need for that? Then it's kind of okay. How do you have, like, if I want cats in the world, how do I have cats where they're not like floating in the air? Are they like anchored to the real world? How are they interacting with it? How are they like, okay, they kind of go and poop in the gravel over there and then you're like, oh my God, I need to clean that up or whatever.

Well, it could be that the cats themselves have their own story that kind of evolve on their own, you know, and even people like the AI characters. I know that Robert's really interested in the notion of even for ar, like there's an AI character that doesn't exist, that's interacting with you and then two AI characters can interact together, that kind of thing.

So there is, and this touches on a little bit of my historic passion of like, if you have a digital character interact with the real world and, and then create these narratives, it needs to understand the world, it needs to interact with it in the same way that you would imagine like a person would, or in my case, like a robot. So then that's how I view like all of the technology that you need to do for augmented reality is the same as robotics. So for these narratives, it's like having a digital character go around and do all these things, except actually like change things and poke things. So I won't poke the microphone. Oh no.

So your characters, your digital characters, you're, you're creating, if they're inside a restaurant, they should behave differently than if they're at a baseball stadium or in a shopping mall or at a playground, right?

Yeah.

So you need to understand a little bit of a lot of the context of the place that you're in.

Yeah.

Tell me how your AI is going to start approaching the world that, that you're building. Because that's how a robot would navigate around. Right. It'll behave differently on grass versus sidewalk.

If you think about it, it's a far more fine grained version of what we're currently doing here at Niantic is. Okay, around parks and everything else there's more grass Pokemon around. Like we're near the water here in the very building. Like there's more water Pokemon that come out there. So it already changes based off of the location, but that's more in like the what city block are you on? And like what is kind of closer around there. Now as technology improves over time, you can imagine having different experiences that are like, it's subtle, but then people are like, oh, I get to experience a whole new bit of Pokemon if I go out to the beach. And then they want to explore differently there. What happens when you get that down even more specific and fine tuned to the area? So like there are people who are interested in like, okay, you have experiences on sidewalks versus the grass versus a flower bed versus like a small little pond. How do you have like interactions that go in between there? Now as you get more fine grained, the complexity increases because it's not just like, okay, is this city park water? But then that's kind of one of the problems that AI kind of helps us get to the next level to.

Talk about our voxel, about infrastructure structure, what's underneath, what's on Your computer screen. When you're programming this, what tools do you use? What are you building here? Are you building internal tools? Are you using open source AI tools? What kind of infrastructure are you running on?

Well, if you want to get into C or Java or everything else, let's get nerdy. So at least to get in, there's probably better people on the AR team to talk about all the specific tools that we work with. But at least we try and build a lot of stuff internally because we're trying to build a complete cohesive platform on top of it. Now the complexity of the technology that you need to build to create this, it's not just like the AR cloud that we can talked about before is just one component here. So one of the things when we came into Niantic, what was just really impressive is just all the other infrastructure that had been built up because if you look at the impact of Pokemon Go, it was just phenomenal, the amount of reach, the amount of people. But then if you think about how much the servers need to handle, how much the anti cheat software to prevent GPS spoofing, like we have some of the most advanced software in there. So actually that was one of the things I began to appreciate. When you're focusing on like the AR cloud versus everything you need to create a real world experience, you're like oh well this is really hard in this domain. And you're just like, well these other like five teams, you're just like oh crap. That's then getting that all to work together. So that's like we're trying to do something that people haven't really done before. Because AR is a, it's different type of computing. It's not humans adapting to computers because before we learned how to type, we learned how to swipe, we learned how to use mouse. Now it's getting computers to adapt to the world. So then how do you have a completely AR focused game? Like one of the cool interesting things that most people don't talk about is to my knowledge like Ingress and Pokemon Go, they're the only game now there's some copies that are out, but they're the only games that you could only do on a mobile phone. You couldn't really do it on a tablet. It didn't make sense, you couldn't do it on a PC. If you think about like Candy Crush, you can do Kinect 3 and other sources, Angry Birds, other types of games, they're like, you can play them different ways. Now there's gaming that can only be done on mobile phones. And like we're interested in what are games that can only be done on AR headsets in the future. Now we're still like, we're not there today, but like that's the progression of where all this will go.

Yeah. How fast is the database going to grow at Pntech? Because you're, as I play Pokemon or say on Harry Potter in a park, I'm grabbing all sorts of things and sending that up to your service. And maybe we should have a second follow up about privacy in this new world because we're going to be aiming cameras and sensors at each other and at things in the world and grabbing things and taking pictures and studying us in a new way that's never really been studied before. Right. What's that database? Is it an Oracle database? Doubtful. Right. It's probably not even a MySQL data. It's probably in. So tell me a little bit about how you're, how you're approaching the infrastructure from, from the data standpoint. What are these systems? Because I bet I would have a similar conversation about autonomous cars with Elon Musk. Right. Building Tesla, he's ingesting. But my Tesla is uploading two gigs per month of data to their servers. Right. And what's, where is that going? What is being done with it and how is it being used?

One of the things about like what we're doing here is it's like there's people who are just like, oh, I want all the data and everything else. That's not what we're trying to do. We're trying to what is the information that you need to have the experience that end users want? So then if you think about it, almost every pokestop or every ingress portal, it's like our users saying, hey, take a picture, I want to play a game here. And that's built up collaboratively with our users. So like kind of the. You're asking the technical specifics on how fast is the database growing? I really don't know the numbers off the top of my head. But then like it's all kind of what do people want? What are the experiences that they want to have? And like we're there to kind of provide the experience and the infrastructure to make that possible if they decide that they want to or not.

I got a quick question. So I don't think this can be done currently, but let's say you have a user that says this is an excellent location for these panoply of characters that you have. Right. And Then a couple of other people or friends, he contacts them and they're like, yeah, I think this is a really great thing we should do. How fast could something be set up so that that location is now a place where all these things congregate? Could it be done on the fly?

Yeah, there's different experiences that can be done on the fly. So then there's. I can't talk too much about what's in the pipeline, but then, like, there's. So it all gets in the specifics on what do you want? So, like, using a specific example of, like, think about ingress portals from, like, a couple years ago. Like, okay, you take a picture. Hey, here's different tags on it. That goes out to the community. People kind of vote and filter all that. Is this actually good? And then kind of, do we foresee that there's any problems with this? Then that can be fed out and pop up as a, hey, here's a new place to play a game or an experience. And that turnaround can just be dependent on how fast the community wants to work with it. So then, like, for future experiences, like you want to. For example, what we were doing at Escher before with multiplayer experiences, then that needs to be built up and synchronized in real time.

Yeah, that's what I mean. The real time aspect of it is really hard.

But that was one of the things that we specialized in or really focused on, because we've shown some videos on how low latency and kind of 5G cellular systems will really help kind of build out the kind of future of these type of systems. Just because the new technology that's enabled, that's not built by us. But then we leverage that and work with these people to kind of create these experiences that. That just matters a heck of a lot.

Yeah, because it makes it feel like the user is in charge of the Their own world and they can build things into existence.

Yeah, yeah, absolutely.

Because we have to wrap up in 2020. How do you think this industry is going to play out? I mean, you don't have to talk about the NTECH, but maybe talk about your vision of where. Because 2020 seems like a pivotal year. Oculus Quest is coming out this year in 2019, and that's gonna reintroduce VR back into the home and into the conversation that we're having. And you guys are coming out with a new game sometime soon. By 2020, we're probably not gonna be wearing glasses as consumers. Maybe at work, we wear them in weird situations. But tell me what you think is gonna happen in 2020 and then maybe what's gonna happen over the next five years after that?

Well, I think if you look at how things will develop over the next couple years, first there's like the current wave of ar. People are now beginning to shift from like, oh, here's the cutting edge of technology versus here's the technology that is needed for this application, and consumers can care about the application. Like, one of the hardest lessons that I learned as a founder coming from MIT was people, people or tech is two steps away from a customer, so then solving the middle step. So right now in ar, there's been a little bit of too much of a focus on, hey, I made a cool tech thing which other people in the AR community really appreciate. So at least I see even the momentum right now is shifting from, okay, what is the newest bar for technology versus what is the application that people want and what is the technology for? So is it product based or is it engineering based? Shift over to product. So I believe by 2020 that shift will have kind of happened more going from the tech culture to more of a, do people want this? I view that kind of like one of the things, like we're investing and like trying to help other companies out to promote the kind of like, we made an investment and Digilens, that's public, trying to speed up the kind of device kind of headset market to make sure that we can be a part of that vision and like, help out. So then that, as you say, might not be in the next two years. I don't know. Like, it's really, really hard technology. I view there will be a kind of consolidation, as there usually is in these markets over the next couple years, into who are the couple players rather than here are the two dozen players and like, who's going to win? We're in a unique position because we have the end application. You of course have Google and Apple pushing Arakit and Aracore, and then there's like two dozen other AR companies that are also chasing after that. So probably just using history as a measure, there will usually be a consolidation around that and then focus.

Irene and I are calling this industry Spatial Computing. How would you define spatial computing to a normal person? And maybe talk to the nerd and talk to the normal person.

Like spatial computing or physical computing. Unsure which buzzword will win in the long run. But at least working with the spatial computing, it's bringing computing into the world so that it connects more to both the context and what is easy for the user so if you think about, like, I believe that computer screens will be the first major, like, mass adoption. And then when you get. Yeah. So for computer screen, it's like, okay, if it's physically mounted here, what would happen and how would our behavior with computers be if you can move it in a whole bunch of different locations all the time. So then as you connect things to the real world, it becomes easier, it becomes faster, and just what will change? Very cool.

We got to give up the room. So thank you very much. It's been an honor meeting you again. And I can't wait to see what your work, how your work changes the world.

Working on it Budget.