SRI (home of Siri) shows us augmented reality binoculars

So what you see here, first of.

All, who are you?

I am Supun Samarasekara. I'm a technical director from the Princeton office in Sri Lanka in the vision and robotics activity there. So in my activity we do a lot of work in augmented reality, specifically for the military. So I'm going to show you some of the technology we've developed that we think is going to become more and more prevalent in the commercial world also.

And so these are augmented reality binoculars.

That is right.

So I have Google Glass on, which does not do augmented reality. A lot of people think Google Glass actually does.

That is correct. You can do some form of it in very simple ways. But if you want an immersive experience, it doesn't really do that yet. We are hoping it will get there. So what it is, is we've taken a military binocular system and taken the guts out. We printed the shell and put our own equipment in there.

This is 3D printed. It's really cool.

This is 3D printed. So just to give you some motivation of what we want to do with it. So in the military world we are using it for military training, but I thought I'll motivate it from some of the commercial applications. So one is for things like tourism. Right. If you walk around, there are so much geotagged information out in the world. What you want to do is to pick up your binocs, look around and start seeing geotags. So basically you can mark up interesting sites to see.

Now this is interesting. Can I tag, If I looked at the Transamerica building, could I tag it through the binocular? And then if other people were using the same binocular, would they see that tag?

Yes.

I can understand why the military would be interested in this.

Okay, and the second part is information sharing, as you were just saying.

Yeah.

So you can mark up a spot, leave a message behind, not just mark up a point, but actually leave a message for a specific person and when the person looks at it, you'll be able to see the message.

Can these tags be put on movable things, by the way, like trucks or people?

Not right now. So I mean the technology, we are just introducing geotags over what is seen. Fundamentally, if you have a way of tracking the person, it can be inserted as things that you see. And in the military world we do that for planes and stuff. I show you a demo. Yeah, that. Quick video of that very shortly.

So will there be a mobile app that goes with this so that I can tell Scope, send this To Scoble.

That is right. The hope would be eventually we have this other VPA technology. You'll just speak to the binocs and say, look, send this to so and so and it'll tag the message and send it right. So no keyboard, nothing hands free.

And the goal in the future is to be able to tag moving objects. So when you're all in the whale watching boat and somebody knows that that's a gray whale mother with a calf.

You can see where the military would be. They've already disclosed that they can put a drone up above us, 17,000ft above us, and watch 17 square miles with moving video down to a resolution of 6 inches. That's up on YouTube public information. And so you can imagine when you have that overhead, then you need somebody on the ground to tag all the things that are on the ground.

That's right. And whatever you're tracking could be tagged and followed on the ground. And I'll show you something similar that we do. So a second area is now what we would like to do is to actually publish an interface where others can plug in their own thing, whether it's a tracker, as you were saying, or for recreational sports. If someone comes up with a bird watching or bird detector, you could plug it in now you can move around your binocs and suddenly it'll go bing, bing, bing, I found a bird. And it will guide you to find the bird. Right.

And it'll probably tell you all sorts of things about the bird.

That's right, exactly.

Now, if using something like Google Goggles, not Google Glasses, but the image search engine, cloud based search engine, you can easily.

Exactly. So hope is that you plug it into the cloud and that will do the processing and feedback the information.

Now, yesterday, where I work, we love the cloud. We love people like you who are going to bring more imagery and more databases into the world.

Now, yesterday, someone was suggesting a very cool idea in this Boston Marathon situation kind of thing. They said, can you plug in a people detector that has a list of people of interest and can someone be just looking around and get say, okay, go look at this person more. Right. So it's an interesting thing. And if you have a framework like this, you should be able to plug in stuff like that.

Face detection is clearly coming. Facebook already bought Face.com, which is an Israeli company, and it worked pretty good.

And the key thing is you need the resolution to detect the face, right? So with the binocs, I mean, a lot of the time if you look at the Webcams that are up on the roofs of buildings, it cannot do the job. Right. So if you have something like this can potentially actually work.

Yeah.

So here's a military application that they use it, use the system for. And this is for what they call the forward observers. These are the guys who are out in the battlefield who mark up targets for, you know, purposes you can all imagine. So what you see on the left here is the kicked up version of your Google glasses, which is true augmented reality glasses a person is wearing and he can see the helicopter flying. This is a virtual helicopter and through the binocs it's sharing the view that is showing the zoomed in view half a mile away of a vehicle it's attacking. Right. So there are multiple people watching this. This is the Google Glass future Google Glass I would say. And that is the Binar crew and they are training on how to call for fire. So this is one of the applications we are using this for.

So are you seeing this as a future competitor to Google Glass?

I actually think Google Glass will want this technology. This is where it will go.

How soon until something like this can fit into something like Google Glass?

I think it's not far. I mean if you look at most of the mobile processing platforms, you get four core, eight core mobile processors coming right now I think it's a matter of time. The sensing is there, the cloud is there and all these things are coming together to really make this a viable system. Yeah, so that's a couple of slides.

It's really interesting seeing wearing the Google Glass product. It's very clear that Google had to make some trade offs for weight, for battery life and for cost.

That's right.

They're clearly aiming this at a $200, $300 price point.

That's right.

If not at retail at first they might charge 500 or you know, so at first. But the cost of this is going to be under $200 and that's really interesting. This stuff would raise the cost or raise the battery or raise the weight and that's going to be further out.

But I think when you get to commercial quantities, all these become more viable. So that's the key point.

What's the image resolution of the sensors that are in the area?

So right now a lot of the stuff we use is limited to 1280, 1024 and it's mainly because the I won't display resolutions and displays that you can go on these things. But that is also technology that's evolving very fast.

That's a lot Sharper than the resolution of my Google Glass.

That's right.

I believe the screen resolution is 320 by 240, which looks pretty sharp, but it's not really good for photos or for live video. That is right.

That is right. So all those will have to.

So you have two of these displays here?

Yes.

The displays, are they 4K displays or.

No, this is very basic AVGA resolution displays. We have high resolution displays. I didn't bring one of those systems here. But this is. What you see here is a Vuzix.

How much does this cost you to build today? It's all custom built.

Yeah. I mean, the parts, I would say is about between 10 to $15,000 and the time invested. So whole augmented reality, we have over $20 million worth of workflow that we've invested in it. So, I mean, what you're seeing here is not through one program. It's over many, many programs. But this one we've been doing for about a year. Right now, what I'm going to show you.

So what would we see if we looked through this?

So the first application. And it's the same image that you're going to see here. What I did is before you came, I set this up. I marked a few things with messages.

So you can, you can see it's live.

Right. And the key idea here is. Let me move this to someplace else. See this line?

Yeah.

It tells you how to get to the message. Right. So if you pick this up and follow the line, you can see where the hello, Robert, message I put there would come up. You're welcome to try it. Otherwise I can't do it for you.

And can I see. I can actually see the image.

Let me pick it up. You can actually pick it up, hold it in your hands and do it.

All right, let me. I'm live. So.

Right. So now follow the line. See when the line becomes short, you add the message.

Yep.

Right. Now I also created a message for you, Shell.

So. So you can see if, if you were over, let's say over that way with another pair, I would be able to see that message. The message is. Is aligned in 3D.

In 3D.

So if I was over.

Across the courtyard, follow the line. See, the line becomes short. So go in the direction of the line.

Go left.

Yeah, I. Oh, that line there. Oh, I see. Okay, Shell, welcome to sr. You went.

A little too far.

Well, I, I read it before it disappeared.

It's still there.

I forgot. You're all seeing what I'm saying.

Yep.

So you can see how bad I am with military level binoculars.

The other point is you want to talk about how the technology is independent.

Of this one pose.

Basically reconstructing the 3D photogrammetry of all the information.

So when you tag that building and you put a message on it for me. Yes. That tag exists in a three dimensional space. So we would see that space even if we walked across.

So these are all getting geo tagged. Now what is very interesting is that is geotagged and we are very precisely geo locating the binoculars. So when you I could take this to the roof up there you'd see the same tags. But not only that, if you put 3D objects you'll see them in the correct perspective from that point. Right. So the next demo I'm going to show you is actually putting some characters down right now if he went up to that troop, if the guy was looking that way, he'd see face first. I would see the side here. So it's actually putting 3D entities, truly 3D geo located in the 3D world. Right. So like what you see here, this is a real backend game engine that's running here and it's working in the real world. So think of the next generation games you're playing in the real world. Yeah.

Google is already playing with 3D walk around games called Ingress and it's pretty interesting that you have to walk. It puts virtual objects on the world and you have to walk into the virtual objects to play with them.

That's right.

That's pretty interesting. So you could see a whole bunch of commercial applications, but it's going to be five to 15 years before we get this kind of technology in our glasses.

That is correct. So this is. But you know, for us, everywhere we see a video camera, we want to augment it. So if you're wearing one in our head, we want to augment it. If it's in the Binox, we want to augment it. If it's a vehicle, we want to augment it.

We should talk about it. Sir.

You want it in the iPhone too?

That's right.

Any smartphone.

Yeah, Anything that has a camera, that's where we want it.

That's a rhyme job that you clip on.

You want that to be on.

HDTV was invented here and so were some of the things like the lines on football fields during tv. Right. You guys bought Sarno for Sarnoff.

That came out of our lab.

Certainly HGTV was done by the Princeton group at sri. And also what was originally the group that did the Lines on the football fields. That was called peb, Princeton Electronic Bulletin Boards. And that came out of Sarnoff, which is now part of SRI as well.

Very cool. So you guys have a. You're not kidding when you say you love to augment every video camera.

We see video, we see augmentation.

How will that change the world when all the cameras are augmented?

See, the thing is, we are really getting to a point. There's so much information and there's information overload. And geospecific access to information is a key thing we believe is useful. You know, you really want information just in time, just in place. So that is what the augmented reality can provide for you.

One of the things I want to do with the Google Glass and it doesn't do it yet, is just say, hey, show me the closest Taco Bell or show me the closest Starbucks. Using this technology, somebody could sit up and just tag a lot of things really, really quickly. Now Google already has a Starbucks tag, but this would be very, very useful for that kind of stuff.

Just to Shell's question for a second, if we had the ability to use augmented reality for all cameras, that could be an organizing principle for all the information in the world. As Supin just said, the information is there. It's contextually located in space for you, which is people think about space a lot. We act differently in our kitchen than we do in the office. So you want different information represented in different ways in different places. And that's the kind of thing that this technology can give you.

Wow, it's really pretty crazy. I don't know if you saw the Israeli company that helped the FBI look through thousands of photos and videos of the Boston bombing, but it compressed all of those images so that they could scan and compare movements of people in real time. It was really pretty amazing technology that I was reading about this morning. You add that with this and you're going to be able to see the world in a completely different way in 5, 10, 15, 20 years.

Right. And again, not just what's there, but what you can put there for the purposes of helping people understand things better or just organize their world.

Yeah, I said the Google Glass is really a shift for Google from an advertising based world to a commerce based world. Because when I walk and we're going to go to a San Francisco Giants game with a Google Glass, the PR team invited us over there to talk about how sports would change with wearable computers. Because Google's not the only one doing these kinds of wearable computers. But you're going to be able to look around the stadium and see where the bathrooms are or where the hot dog stand is, or where your friends are in the stadium. I can already do that with the Oakley glasses.

So just one more thing. If you. If you want a wild idea. So if you go back in history, like 500 years, human memory was something that was super important. And when you went to university, they would teach you how to memorize things. And one of the ways they would do that is spatially. So they had this concept of a memory palace where you'd imagine yourself walking through rooms, and the things that you were visualizing just in your imagination would help you remember things. So now we're going to be able to do that in augmented reality. It will be an augmentation of human memory and cognition.

Pretty interesting stuff. We're going to put our task lists on that building.

It's not a joke.

I know it's not a joke because I'm actually going to apply, like, if I need to pick up a power outlet, I need to pick that up at the hardware store. Right. So I'm going to put it on the hardware store. And that way, when I walk within 100 yards of the hardware store, up comes my cast list.

Absolutely. The list doesn't have to be text. It can be exactly that item that you need to pick up.

So you got the characters up.

Yeah. So, yeah. While you were chatting there. So this.

Oh, you mean a character, like a human.

Right.

So you put a fake human, a virtual human.

Yeah. So now here I don't have the game engine, otherwise they could be running around, too. So if you look, there are some special VIPs here. So who's in a Humvee, and there's a, you know, guy at the entrance there. And I believe there should be another guy around here somewhere, hanging out in.

The trees.

Who'S guarding the car.

Get him, Rocky.

Now, a key thing about the technology we do is also. See, I can shake this, and it's shaking along. My camera's shaking, too, but it's like rock solid with the ground. Right. And to do this at 1km, 2km away is not an easy thing to do. So that is one of the key techniques.

How much code did you guys have to do that? Can you give me a hint of what the algorithms are doing to keep that image so steady?

Yeah. So we track features, but what is interesting here is we track features on a wide field as well as a narrow field, and we combine it with the inertial sensors, like imu, gps. And we combine all of that to get to that rock solid stability.

And this probably partly came out of the work you guys did with the football lines. Right.

It's all a continuum in a sense.

Because the early football lines, they had to put markers on the field, and now that thing works without any marker field.

Right. Now the key thing between the football field and the stuff you see here and what we do on the helmet systems is even what they do today on the football field, the cameras doesn't move a lot. It's very much at some fixed locations. They're panning around here. You want total freedom in moving around. Right. So you should be able to walk out of the building, come into the building, go to your grocery store, and it should keep track.

And this is all tied with cloud computing, because if Rocky's across the street using his goggles.

That's right.

He's gonna see that guy from his point of view.

Exactly, exactly.

So we could really play new kinds of games.

Absolutely.

Paintball is dead. We're gonna be able to have virtual shootouts in the street. And people are like, what are you guys doing? We're out of games.

You can play virtual a paintball in your home without messing it up.

That's pretty cool stuff. Wow, that's pretty mind blowing. So we're pretty close to that with the Google Glass. Battery is a problem because keeping it on my battery only gets an hour and a half, but I'm carrying a Mophie pack in my pocket, which extends the battery by seven hours or so. So that's really cool.

Yeah, so that's. I mean, lot of tech challenges like battery, communication bandwidth, all these things, you know, but all these are things that will get addressed, you know, going forward. So well.

Thank you so much. On sri, where do we follow your work?

So we are in the principle.

What would I search Google for?

If you search vision technologies under sri, you will find the activity in our group. Probably my name will show up in a lot of the references too.

Great. Thank you so much. This is really awesome. And I hope it escapes the military soon so we can use it for shooting each other. In the shooting. Rocky's like, no, I'm not going to shoot you. You know, you're going to. When you get Google Glasses, you want to put one in my head? Come on, man. Thank you so much.