Augmented reality on mobile arrives at Qualcomm

Who are you?

So my name is Jay Wright. I am vice President of Business Development.

At Qualcomm, and we're talking about augmented reality. And where are we today?

Where are we? I think we're probably what I would describe as the second phase of augmented reality for mobile devices. I would characterize.

No, but what's the physical space we're in?

We're in the Qualcomm Museum today in our headquarters building at our main campus in San Diego.

It's really cool. And one of the first cell phones is behind me. And it's big.

It is big. They all start out big and end up smaller, faster and cheaper.

And you're taking advantage of that trend, right, in augmented. So we're going to talk about augmented reality and where the world's going with these glasses and wearable computing and. Well, tell me where the world is going, because you guys build the chip sets inside almost every mobile phone out there, right?

We do. We do. We provide chipsets for quite a number of models of phones and tablets and other types of devices, too, which are out here. Yeah, we've got a bunch of examples that we'll show you today. But I think where the world's going, at least as far as phones are considered, is more toward this context awareness concept that I know you've been following a lot. And we've got this vision at Qualcomm of a digital sixth sense, where we're able to do sensing all the time and increasingly all the time, so we can take different information and find out what you're doing and help filter information that you're receiving.

And so to people, when they hear that, first of all, they might be freaked out. I call it the freaky line. What do you mean you're tracking me all the time? Well, tell me what you're able to track right now on a mobile phone. If it's in my pocket, just tell me what kind of sensors.

So location is something that can be tracked. Certain gestures can certainly be tracked by sensors that are there. And then with the camera, we can actually recognize certain things in the environment. And that's what Vuforia and our augmented reality platform are all about. Yeah.

And where is it going in the future? Because I was just at Oakley and they talked about having cameras on goggles that could both look forward and capture what you're experiencing, like skiing a run, but also look at your eye and track your eye where you're looking. So you can build new kinds of user interfaces, right?

Absolutely. And it's probably that Class of device or that class of glasses that at some point will displace this phone or primary device that we carry around in our pocket. A large number of those same experiences could probably be reproduced in glasses if we had that eye tracking capability and the ability to recognize everything in our environment. Wow. We can't have our hands free.

And so you've been working on the recognizing things in our environment. What's that technology called?

Computer vision, generically. But the software platform that we've built for developers is called Vuforia. And Vuforia is all about making devices and apps. See, so we're essentially turning the camera on your phone into a digital eye.

Do you have any examples of that?

I do. We've got a ton of examples.

Let's walk around and you can show me a few.

Sure. So we're seeing applications and today, Robert, We've got over 2,000 applications that have been done by third party developers.

That's already today. This is not in the future.

This is not in the future.

You don't need the iPad7 or anything like that.

We do not. We're going to be showing these on commercial devices and these are commercial applications. Although I do have some R and D demos that are here too. So the bulk of the applications that we're seeing are marketers and they are using computer vision and augmented reality to create engagement with their advertising and with their products.

And that's a big deal. Oakley is thinking about how to use these technologies in retail stores. Luxottica, who owns Oakley, has Sunglass Hut has all these retail channels to sell eyewear and stuff like that. And so they're building in their labs right now these interactive retail experiences where you walk in with your cell phone and you can do things, right?

Yes.

So what can you do?

Yes. So I'll show you an example. I actually have a retail example after this. This is one of the examples that's been done for the publishing industry. So what we're seeing now is a large number of applications that are companion applications for print publications. So this happens to be one that works with Maxim magazine. So there's an application called Maxim Motion. And if you have Maxim Motion, it will work on certain pages of Maxim magazine. So you'll see here, for example, as I point the device at this magazine, this image on the COVID will actually come to life. And they've done this video in a pretty creative way. So it looks like she's actually jumping off the page. Okay. So as you look here, you can see almost seamlessly it was recognized. And now there's a video that's being played directly on top of the page. And so this will continue. This experience can happen throughout the magazine, and that depending on the publisher or what the advertiser wants to do, there can be some kind of call to action directly inside the magazine. There can be a purchase capability directly inside the magazine, but it's giving the publisher a new way to turn physical print into an interactive medium.

Now, technically, what happened there, it turned the image, the pixels of the image, and what did it do in the computer? What happened?

So what happened is we initially recognized this image. So this application, ahead of time, has a database of the images it knows how to recognize. That database can live on the device. And if you've got a million or more images or a huge database that you can't manage on the device, you can also put it on the cloud. So what we're doing is processing every camera frame that comes off that camera sensor. We're comparing it to images that we have in the database or the cloud. If we find a match, then we're loading the content, or the app developer is loading the content that should be associated with it. And then Vuforia tracks that image 30 times a second to. To see where it is in the camera frame. So the graphics can be drawn in the appropriate place. And that's what really creates that realistic effect, as if the content is really in the real world.

That's really cool. All right, what else can you do with this stuff?

So let's go ahead and show another educational example here. And I am going to switch devices. Apparently, this is not on this device here. So this is a children's book, and like many children's books, it has activities inside of it. And this one is called Rocks in My Socks. And it's about teaching kids to deal with adversity. You know, sometimes things are a little tough. I've got rocks in my socks, and I've just got to learn how to deal with it. So as I cruise through, you'll see there's some. Some exercises. So, for example, this one is a dinosaur kind of matching game. And you'll see now when I put my device on top, I'm going to get a completely different experience on top of the book that allows me to play a different matching game. So we can see here all our different dinosaurs are hanging out. And I'm actually a little embarrassed to admit that this isn't that easy. And I will make mistakes on here, but I'm going to go ahead and Try Stegosaurus. Try again. Okay, let's try him. Stegosaurus. Nice job. Yep. So you get the idea. I'll go ahead and quit while I'm ahead.

Now, there's going to be a whole new range of toys. There's a company called Toy Talk, a startup that just got $16 million of funding started by the former CTO of Pixar. And he's going to use this vision on actual physical toys. So you're going to hold your Thomas the Tank Engine in front of a camera on the iPad, and it's going to say, hey, you know, welcome, and I think we probably nice Thomas the Tank Engine toy. Right.

You absolutely. So toys have this physical play experience, and now there will be another digital play experience on top of them, which is great for a toy manufacturer because it's extending the life of the toy. But there's also a new revenue opportunity because now that toy manufacturer can sell digital content inside the app that works with the physical toy.

Now, this technology works great when a developer has actually put this page in the database and taken a picture of it. And what happens to that image in the database? What, you know, tell me about how it's processed is, I assume that's being processed and cut up into pieces so it knows what it's looking at.

So that that database does not contain all the bits from the original jpeg. What's actually stored in the database is a munged version of that JPEG that you can think of as an image signature, but it's basically a collection of what we call feature points, which are small little areas in the ID image that have a high variance in contrast. And it's the number and distribution of those feature points that forms that signature. So what happens when we're actually doing the matching and looking around is each frame we're looking at the image using that same algorithm to extract the feature points, create a signature of what we're looking at, and compare it to the signatures in the database.

Now, I assume that in your database there's only a few hundred thousand or maybe a few million images. That's not enough for you to walk down the street and get a hit on everything you're looking at. Right. I assume that someday we're going to be in a world where you will get a hit on literally everything you're looking at. How do you get to that world? Is this technology scalable enough to deal with a world like that where you fingerprint everything in the world?

I think so. I think so. And Qualcomm's working on this technology. Other folks that are in the search business are also looking at this technology and shown large numbers of images can be searched and, and visually identified. But what we're trying to do is work with developers that have one particular brand or application that they're trying to associate with a fixed set of products or pages or even products that might show up in the store.

Yeah, because this is going to take to build a company that actually recognizes everything in the world. You'll need to build a lot of data center space or take a lot of space on Rackspace Cloud, which is expensive.

You'll need a lot of Rackspace.

Excellent. That's what we like to hear.

But you brought up a good point and it's one. We're talking about the user experience. And you can create an application today and try and make it recognize everything in the world. But that's sort of the boil the ocean approach. And the reality is I can't recognize enough things really for it to be compelling for a user to continually use. So what we're trying to do is enable application developers that work with a fixed set of objects that will work very reliably. So we know there will be a good experience and the developer and brand knows there's going to be a good experience with it.

Yeah. Well, I assume the guy who started Keyhole, which now is Google Earth, he said by 2020 we're going to have perfect vision. He thinks that by 2020 we're actually going to see this move out of the domain specific instances to the world where we're just going to walk down the street with our glasses and it's going to hit on a lot of things, both because of our GPS, but also because of the camera on the glasses.

I absolutely believe that. I think all these things will converge. All these databases that today are in a single app will be unified in some fashion and the right experience will happen when I see the right thing. And hopefully we're able to filter that in such a way that people aren't overwhelmed by things they're not interested in. But I'd like to show you a demo on the toy front. This is actually one from the labs. We've shown it a couple times before. But you mentioned the ability to augment physical toys. And what we're going to show you is actually state of the art for 3D object recognition. And it's something we worked on with the Sesame Workshop. And as you probably know, Sesame is very big on using new technology to help children learn and what we've done is adopted a toy set concept. And with this toy set, there's actually different mats. And these mats are envisioned to be different rooms in the house. So in this case, I think this is Bert and Ernie's bathroom floor. This is their living room. And then these figurines are different things that I might find in a playset. Now, they happen to be a little psychedelic looking so that we can make some of the computer vision work. But I'm going to go ahead and show you what this looks like now when we add the tablet. So I'm going to go ahead and bring up the sesame application. And again, this is a prototype, but it's going to show you exactly what's possible with toys. So I'm going to go ahead and make sure I've got the flash on. So the first thing that you're going to notice is now when I'm looking through the tablet, I don't just see the floor, but I see walls. And we're actually in Bert and Ernie's living room. And I actually know that because if I go really close, I'll see Bert and Ernie on the wall. But the magic happens when I take the physical toy and put it inside the play environment. So let's go ahead and put Ernie inside. That's me, your old pal, Ernie. Hey, put another Sesame street friend in here with me. So what just happened there is we recognized a 3D object. We took graphics and a model of Ernie, the same one that was used for this 3D print of the toy, to create animation that fully envelops the physical toy. And the effect here is that Ernie has come to life. And this was really exciting to see kids react to. So now Ernie has said, hey, why don't you put Bert in there? And what we're doing is encouraging the child to explore. So I'm going to go ahead and try and do this. So we'll go ahead and now Bert's been introduced into the scene. Bert's a little skinnier.

Yep.

He gets hard to recognize sometimes. Oh, look, it's Bert. Hiya, Bert. Oh, hi, Ernie. It's great to see you. Nice room. Put something in the room that Bert and I can watch. Go ahead. And then we can take the concept further. So I'm going to go ahead and put the TV in the room. Wow, a tv. I hope it gets the pigeon channel. And you'll see we're actually playing a video on top of the tv. Yep. So this wouldn't have to be a child holding the tablet necessarily the whole time and trying to steady. This could be something that could be sitting down in a rack. But it shows you the type of experience that's possible, right? The actual physical toys coming to life and creating a new and more beneficial learning experience on top of them. And then technology wise, this is also first of its kind. What's happening here is we're recognizing 3D objects from any angle and we're doing more than one at a time. So most of the demos you'll see when you see AR technology will be maybe one or two things, but these are multiple objects all happening simultaneously. And this is actually on a dual, dual core Snapdragon S3 processor.

Disney owns the patent to pass electrical signals through the paint on top of the tablet itself.

Seen that? The Cars app. Yeah.

Well, I interviewed the guy who invented that in Sweden and it's pretty ingenious the way he figured out that you can pass electricity through paint, through conductive paint onto the tablet surface where there's a sensor.

It's really cool.

Yet another sensor in our lives.

It's got to be good to be a kid today. Actually, one of the surprising things and kind of funny stories is the first time we saw that work, we thought, oh my gosh, toys are coming alive. How amazing is that? But when you tell a three year old, hey, hold this thing up and the toy is going to come alive, alive, of course it will. There's no magic or wonder. It's just the expectation. Right. Things just do that really funny. So you mentioned retail, and retail is a big opportunity for augmented reality. One of the things we can do is create a better shopping experience in the store. So as we probably know and have experienced, shopping online for things is a little bit better than in the store. Many times I can see all the colors that something comes in, all the sizes.

Well, you have infinite inventory. Amazon proved this to us, right, that they could sell many, many more books and give us much deeper choice than even the biggest Barnes and Noble store that we could walk into.

That's right. So we worked with American Apparel on this demo right here. And the idea was they wanted to create an experience inside the store that was comparable to what you had online. I want to be able to find all the sizes, all the colors, and get comments and reviews from products while I'm shopping in the store. So what I've done right here is mock up a very simple environment, as you might see in American Apparel. You'll see some clothes that are laying around and then you would also See, see here, these little shelf tags that go next to them. Yep. And so what we can do now is point our device. And I've run this American Apparel application. We can point our device at the tag. And we'll see here. We're using the cloud now, so this database is stored in the cloud and as it scans that image, you'll see the image actually jumped out of the camera view and became part of the user interface of my application. So you'll see here also, the price is now visible on the app, whereas the price was not visible there because it doesn't make sense to print prices there that change all the time. And then I also have the ability to see all the colors that this comes in. I have the ability to look at reviews very simply and easily.

Coming soon. By the way, there's some startups that you step into a booth with Xbox Kinect sensors.

Yes.

They can scan your body to plus or minus a quarter inch.

Yes.

And they can put that cloth on your body.

Yeah. The virtual dressing room applications, as we call them. Love those. I need to get that in my living room because then I can officially never go to the mall.

You can actually do it with one Kinect sensor, which is now a hundred dollars. Right. So it's coming, coming soon. And we were at Autodesk last week and they showed us that you can scan somebody's face very exactly. And actually make a virtual model of somebody. It's really crazy.

Real exciting stuff.

The retail world is about to change a lot. Oakley showed me some concepts that are confidential right now, but the interactivity in the retail world is going to be pretty crazy. So what else have you got?

I think it will. I've got another educational example that I'd like to show, and this one that was actually done for the College of Education and the University of Illinois, and they put this application together for the School of Nursing. And what they were trying to address is a shortage of cadavers. There's not enough cadavers for people to use to learn about the body. So this target is meant to be the surface, where the cadaver would be or the virtual cadaver. In this case, we've got a smaller version and the actual version where students would be using this. This we printed out full size, so that would be head to toe. But what you'll see here is when we point, when we point down, we get a very accurate model of the human body. So these guys spent a lot of time, as you can tell, with the 3D graphics, to get this really, really good. And I can turn off different layers, if you will. So the muscular system, skeletal system, I can turn on and off. I can make skin appear so I can see actual skin on the body. But what's neat here and what augmented reality is bringing to the experience is this notion of physical scale. Because I'm looking at something and I'm moving along and I get the sense of where I am on the body and then the ability to look at it from any angle.

If you watch the Wright Ted video, you can actually 3D print a lot of the body now too. So we're going to have virtual cadavers where you can actually have a physical one standing there pretty soon. Pretty crazy stuff. That's cool. So it's using the same image tracking technology that the magazine used. It just is looking at pieces of this image.

So everything I've shown you is now a commercial application, with the exception of the toy demo, which is still a prototype. But we're working with some toy folks on some pretty amazing things for the future.

I bet my friend's starting Toy Talk and what he's been talking about is pretty mind blowing. So it's going to be a lot of fun. So what's over here?

You got a little village here. This is another R and D demo. And this is useful when I want to create an AR experience that actually responds to objects and ele in my environment. So wouldn't it be fun if I could have little characters that could run across my desk and bounce into my computer when they run into it? Or a child to have an experience in their room where characters run around and jump on and off the bed? That takes a little bit different type of technology. And the name for the technology is called slam and it stands for simultaneous localization and mapping. And it actually comes from robotics. Robotics needed to do this stuff so robots didn't bump into things. But what it does is it allows us to build in real time a 3D model of the environment. So you'll see here as I'm moving the device around and we're in a diagnostic view here to show you what's going on. But you'll see these yellow and red points appearing. Those are feature points in the environment that identified as common points. And when I get a certain level of confidence in those points, you'll see that we're dynamically creating a 3D mesh represented by those green lines around it.

You can see we're pushing the processor pretty hard on these chipsets. That's why you need Multi core and faster processor.

That's why you need multi core and it's also why you need to really work on these algorithms, these computer vision algorithms from the ground up and redo them for mobile. So that's one of the things that we've really been working hard on. So you'll see here now I've got Santa sitting in the middle and if we come down a little bit lower, Robert, and we want to prove that this is 3D, I'm going to make Santa come and jump on the top of that building and now he'll come jump down and run around. Yeah. So really, really fun. And I think there's going to be a lot of really interesting experiences for kids and others where now we can bring these digital experiences into real world environments. And of course, this is all the same technology we'll need and use in glasses when those are available too. Very cool. Yeah, it's a lot of fun.

Where do we track your work? Do you have a blog or do you have a place that we can watch what Qualcomm's doing in terms of augmented reality?

Yes, absolutely. So the place to go today is our developer website. So that would be developer.qualcomm.com vuforia and we'll actually have a new web presence that'll be launching in the very new future. But you'll be able to get links to it from the one I just mentioned.

So this, the Vuforia is available as an SDK. Is that available on what, Android and iPhone or just Android?

Android and iOS. Okay, so it's available for both. It's freely available. You can build applications and distribute applications. Absolutely. For free. The cloud Reco service. If you want to use databases that are in the cloud, there will be a fee for that. But there's a certain level of usage that you can use for development. Some limited commercial scenarios. The free of charge.

Very cool. Well, thank you so much for giving me a little demo this.

Glad you enjoyed it. Thanks.