Augmented Reality adds "fake actors" into human scenes (part of SRI International Tour)

So what are we seeing here?

We're about to see augmented reality. Augmented reality has been a major theme for SRI for many, many years. And in fact in Princeton, some of our earliest work was for called Princeton Electronic Bulletin Boards, which created the kind of augmented reality that you see in football games, advertisements on walls, things of that nature. So let me introduce you to the team that helped make that happen. This is Teddy Kumar. Teddy is the director of that activity, Sukun, who's going to show you a demo today. Jayan. And also from which all good things happen. This is Bill Mark, he's the vice president of the Information Computing Sciences division where this by the way, and also other type of ventures like Siri and Nuance and others have come from that division.

Very cool.

Cool. Well, let's sit down and see a demo.

So what is this that you're wearing?

So basically this is a helmet with a camera on top.

Right.

And what we are doing is we are going to take the video from the camera and I'm seeing it here, but in addition to the just the video, we are going to augment it with other stuff. Okay, that's not really there. All right, so Teddy, do you want to set it off?

And you just have a standard old.

Laptop in here, Teddy?

Yeah, yeah, this is a standard Dell laptop. So this is just a.

And what do you.

This one?

No.

So just so you know, what we're setting up here is a mechanism whereby is looking at the environment. They're trying to reset the environment to say this is the environment within which we will do augmented reality. And that's why there's a space over here. And what Supun and Teddy are trying to do here is say, okay, once I've set it up, we can insert artificial characters into the environment. And then once you have the artificial characters into the environment, the big challenge is, is how do you move around the environment with those objects or those people looking as real as they possibly can within that environment?

What would this be for? Like a training exercise or something like that?

Yeah. So let me explain. The background of this is originally US Government defense. And so what's been done is people have put together simulations of what they might experience, such as in Iraq or Afghanistan. And in fact, some of the equipment that you're about to see are people wearing military equipment. In these simulations you get the full effect, as if in this environment another individual is here, such as say a terrorist, and that terrorist is moving around and so forth. And as long as you're wearing these glasses, you will not only see the environment through those glasses, just as it is right now, but you also see what's inserted into there.

And.

And you can therefore see a virtual environment that includes the terrace, for example, together in this room.

So what did the technology have to do? Did it have to map this room so that it can effectively put a virtual character here?

Yeah. So what is this?

So what we do is, with the camera, we can move around and build a map of the environment. And then as you wear the camera, then you move around, it automatically matches to that. And then we have a game engine which can place characters in the scene and you can interact with them.

And do you think this can be privatized in any way?

Okay, so exactly the right question. And as we'll talk about, certainly for.

Training, like firefighting or police or military, that kind of stuff. Or even customer support, maybe.

Exactly.

Or in fact, a gate agent at United Airlines or something. And have to deal with somebody.

Exactly.

Or you want to see yourself, you know, wearing a new set of clothes, or you want to play golf with Tiger woods, or you want to go to the moon.

But we're already seeing some of the. Ebay just came out with an app that lets you wear different glasses and see what, like, a different Oakley glass would look like on your face.

Exactly. So there's a huge amount of commercial aspects of this.

The big app, the biggest app is, of course, it frees you from the tyranny of being in front of your tv. Now you can play video games in your backyard. You can go anywhere, and.

Absolutely. Now I see a Microsoft Connect system.

What are you using that for? That's not for this demonstration. That's for the next one. In this demonstration, everything which we're doing is just running on this backpack with a laptop. And you'll be able to wear this later if you'd like to, and you'll see it on your display. Okay. And exactly what he's seeing, you'll see on that monitor. All right, cool. We'll have it up in a second or two.

No problem. How much code is running on the backpack?

It's a lot of code. It's a lot of processing, which we've done. We demonstrated this system for the military, for the marines at Camp Pendleton, and for the army at Fort Benning. And these are projects which we're doing with them for helping people train. Right now, when they train, they have to hire a lot of role players, and it's costly to hire a lot of role players. And this allows you to have the computer generate the role players also allows you to train anywhere because whatever you train with, you wear. And then you can train right near the battlefield instead of coming back home or at the home station.

How many different role players can it show at one time?

Like in this.

Can I put 10 terrorists in this room?

Yeah, you can have 10. And you can also have civilians. Okay, right.

And tell me about the technology that you guys developed and what was really difficult about.

So the big thing is when you do this technology, you're wearing this on head mounted displays and your heads can move really rapidly. Right. And the objects must appear very static. So you see the 10 yard line in football and it's augmented, it augments the scene. The camera can pan around and you see the 10 yard line of static. Now we're doing the same thing with camera. Then you can walk around, you can walk around the object in 3D. Look at it from the front, look at it from the back and you can see that it looks very static with respect to it. The other thing which we're not showing today over here is we also do occlusion reasoning. So you have the weatherman, the weatherman stands in front of the green screen and now you can see the weather behind the weatherman. Here we do the same thing, but with stereo cameras. We're just showing you a monocular version today, but we also have back home stereo version and that allows you to do occlusion reasoning. The aftar can go in front of people and include them, or people go in front of the avatar, include them. Interesting.

What's the complexity of an object that can be in a room? Can you train somebody to do bomb disposal?

That's right.

That's the motivation by which the military is funding us for how to handle when IDs happen, how to do those kind of situations.

So just to be clear, almost all of SRI innovations start with something that's government funded and clearly there's a great use in the government. But that's not our ultimate market. Our ultimate market is obviously to make those devices. No laptop on your back, you know, no helmet on your head, but instead just a pair of glasses. Perhaps going back wirelessly to a server and being able to instantaneously see these views.

The technology that developed the yellow line in football games that we see, what was that technology originally developed for?

That was originally developed.

It was originally developed for baseball games actually the first time we ever showed.

So we didn't have a military start?

No, that's right.

It was started by an ex vice president of Sarnoff at that time. And the first time we ever did it was in 1994 for the Baltimore Orioles game. It was the first game on April 1st, and I was there at that stadium. We were setting up the technology to demonstrate it. And I remember having to argue with the groundskeeper because I wanted to put some black tape on the baseboard to make the lines appear better. And that was a very proud thing for the groundskeeper to let me put the black tape. But we did, and that was the first time we ever demonstrated it.

But again, the original technology, actually, that use of the technology was not done for the military, but the original technology for looking at stabilization, for looking at target recognition in terms of what the image is that you're seeing and inserting information and stabilize.

That's right. But after that, we didn't see. When you see that 10 yard line in football, there's no black tape or anything. That was right in the beginning. When we started, we were just trying to accentuate the lines. But then the technology improved and now you've been seeing the 10 yard line for years. Oh, yeah, that's right. But this is amazing now when you'll see this, because now in a stadium, the cameras are fixed, but now you can move around freely. Okay. You can wear it. You can move around freely. Wow.

Okay, so I think where are we.