Jason Jerald (NextGen Interactions) - Interacting with Virtual Humans at AWE 2014

Next up, we have Jason Jerald, founder and president of Next Gen Interactions. All right. As Rocco mentioned, my name is Jason Jerald, and I'm founder and principal consultant at Next Gen Interactions. And today, I'll be talking about virtual humans, some things I've worked on, as well as some things my colleagues have worked on. Well, first, since this is an augmented reality conference, I thought I'd start off with the other realities, right? On the left, we have something called real reality, which is interesting in itself. All the— so there's a broad spectrum all the way to the right of fully immersive virtual reality using systems, head-mounted displays like the Oculus Rift. And you can kind of see what I call mixed reality, which really is augmented reality in its different forms. Starting from that left side, including me embarrassing Ron Azuma sitting in the crowd here, that gave a great speech this morning on augmented reality, one of the pioneers in the field, as well as some other examples of different forms of mixed reality. So today I'm going to be starting on the right, kind of talking about virtual reality, that's more my background, and evolve into discussing how virtual humans apply to these different types of of mixed realities. And so I thought I'd start with an example of a truly fully immersive virtual reality experience versus an augmented reality experience. This is the Virtuix omnidirectional treadmill, where you can literally turn in 360 degrees and run in any direction. And this was a game I created that was used with the Virtuix Omni on ABC's Shark Tank. Put on the virtual reality glasses. Oh, cool. This is right up your alley. The Omni enables you to walk, run, jump, and crouch in 360 degrees and put your mind and your body into the video game to transport you into a different world. And so there's some virtual robots in that video because virtual robots are a lot easier to do than humans. I forgot to ask you at the beginning of that clip is to see if you can determine what the virtual human was in that video, which is a trick question. Of course, the virtual human is your own representation in that video. And what was not shown in that video, the player didn't happen to look down so that he could see his own feet. He could see his own body. And that really, that self-embodied avatar or virtual human, that's really really something surreal that, you know, allows you to look down and you see a virtual representation of yourself. And even with a different version we have with simple hand tracking, an embodied avatar, that can be extremely compelling. And it's very important for— there's been a lot of discussion of presence lately in virtual reality. And oh, also I wanted to mention the blue arrow there. What is that? Well, that's sort of a real-world digital cue. In the virtual world that shows which direction the real— something in front of you in the real physical world. And that's sort of an inverse of augmented reality, because we're representing the real world by something that's digital. And that can help reduce simulator sickness because we have sort of the stabilized cue, assuming low latency in a well-calibrated environment. That helps reduce that latency— I'm sorry, that simulator sickness by providing that stable cue. cue in the environment. It kind of keeps you from getting lost as well. Now, of course, if you have your own body, you can also put in another player. For example, here we have a multiplayer game that I created, and that creates that sense of another type of presence called social presence. And here, because we're tracking the head and the hands as well, we can do really compelling multiplayer experiences. For example, tracking the head. There was a question about eye contact. Something as simple as tracking the head, because the camera isn't offset from where the person is looking, you get these really kind of creepy feelings when someone looks at you in the eye. It feels very different even though the eyes are not being tracked. You can think of something like Sesame Street. You notice most of those characters, the eyes don't actually move. It's really only maybe the hands moving. The arms and the head. And so just by tracking those 3 points, this can be very, very compelling. And, you know, we can use inverse kinematics. We can move the legs by walking forward and such. And that body language allows us for very natural and subconscious communication. Here I have my colleague Paul that is, you know, he's got the gun. I'm pointing my gun at him. So he just, you know, he raises his hands up and says— Hey. He says, I surrender. He doesn't have to think about, you know, oh, which button was the surrender button? No, he just, he does that. He might not even be thinking about what he's doing. And so that's really compelling when you start tracking the body for these virtual humans or avatars. Here's another example that I worked with, I'm working on with Digital Art Forms and Sixth Sense Entertainment. This is a National Institutes of Health funded grant. Teaching kids about their brains. And we're piloting this in a school in North Carolina here in the fall. And this first title is called Cure Fred. And that allows us to interact with a patient, Fred, by poking that virtual human and seeing different parts of his brain light up. Or the inverse of that, if we poke around in his brain, we can see his behavior or perhaps his body moving in different ways. Still very early in development, but it kind of gives you an idea of what we can do. do with virtual humans in education. And also because the hands are tracked, it's, you know, that not just teaching with visual and auditory cues, but that kinesthetic or that proprioception sense that's extremely important, especially for some physical learners. And here we're starting to move towards augmented reality. And my point here is we can take a lot of the same content Even though the hardware and sort of the systems issues are very different in augmented reality, a lot of times we can take that content that we're using in virtual reality and plug that fairly easy into augmented reality. So this is a standard tablet-based augmented reality demo with a virtual human I put on top of my business card here. Hi, my name is Joe, and I work for NextGen. I'm a simple example of augmented reality technology. My colleague Jason and I would love to get your feedback and your ideas of how such technology might apply to themes— Okay. Now I'll move on to some work from my colleagues at the University of Florida, Ben Locke and Andrew Raab. I'm sure some of you know. They're using virtual humans for nurse education. And what they're doing is they have these, you know, simple displays of the virtual humans placed in the physical environment. And so they can roll those virtual humans around and actually place them anywhere in the environment. And the interesting thing here, you see the legs down there are actual physical legs controlled by, you know, a physical robotic device. And by combining them together, that creates this real sense of physicality in that space. And Ben Locke has actually spun off a company called Shadow Health that's doing doing very well, one of the few augmented reality companies that's actually starting to become profitable. And so a great example of, you know, taking a simple display, putting it in the physical environment, and it works. We don't have to worry about, you know, so much about latency and calibration and such in the system. Here's another example of making augmented reality physical. This is out of USC ICT, and by using projectors, with an approximate physical surface, you can really minimize a lot of the cue conflicts apparent in many augmented reality as well as virtual reality systems. And here, this is just some retroreflective material similar to the CastAR system. And you can see on the right side with the projector on the face, it's really compelling even though the surface isn't this perfect planar surface. And so my point here is that I think in the future, we may be moving— augmented virtual reality might be moving towards a physical medium. It's not something that's just virtual and you reach out and touch it, it's not really there. But this really can help provide a compelling, realistic experience. Another great example of— out of USC ICT— of making the augmented reality or virtual reality more physical is by combining physical reality with simple displays. And so this is really one of my favorite examples of placing virtual humans in a physical environment. See if it will play. This is no typical shootout. Some serious science and storytelling make the Gunslinger experience unique. Let's learn what allows this simulated showdown to go down. Now you all listen up. With computer vision and natural language understanding, Where are you from? Kentucky. I moved here to get my career started. The characters can track who is being spoken to. How about you? Where are you from? I moved here from Gatlin, Tennessee many, many years ago. This sophisticated natural language system lets the characters be interrupted. Shut the hell up right now! You shut the hell up! You be quiet, you idiot! So you can see how that can be very compelling Just by looking at simple things, such as which direction is the user looking? Is he looking at the virtual human? Then he can respond appropriately. Another example, yet again out of USC ICT, is moving beyond the obvious input metaphors. What if we can sense gaze, which way we're looking, or if we're simply leaning back or leaning forward? We can really respond appropriately using this sort of virtual therapy here. And I don't have time to go into it. I have a question. for that video, as we're running out of time. But I can certainly show you that afterwards. This morning, Tish Hoot gave the keynote, and she talked about the virtual reality neurogaming panel I had the pleasure of moderating a couple weeks ago here in San Francisco. And we had Ana Marquez on the panel, and she showed this headset here. It has a head-mounted display, and it has EEG sensors combined with that. Now, what's interesting about this is she also— this headset also does electrical stimulation to the brain. A little bit scary, but certainly interesting of what we might be able to do with that. For this is, you know, not only output from the system, but input into the brain. And so they've done some similar work in magnetic stimulation of actually conveying signals to another human being. That is brain-to-brain interfaces. Very simple. It's not, you know, reading minds and such, but— Kind of gives you a teaser of things that may be possible in the future. Richard Boyd and others have also worked in this area with DARPA. And it's a pretty exciting time, as mentioned in the keynote this morning, of how those technologies might combine together. Okay. With that, I'd like to sum up. I think a common theme at the conference has been talking about what is the story? What is the emotion of these technologies? Today, we're really selling the technology and the features, but I think we really need to bring this out to the masses. We need to start selling the stories and the emotions and figure out how to best engage through emotional storytelling. And I think virtual humans are a really compelling way to do that, with those virtual humans with realistic interactions and behaviors as well, and of course sensing real humans. Right? And interpreting what those human actions are so that the system can respond appropriately. One last thing I'd like to mention is that I'm working with the SIGGRAPH conference committee. And we're working on making augmented reality and virtual reality a primary theme of the conference. We have a panel of leading experts in the field. We have Michael Abrash of Oculus, formerly of Valve. We have Henry Fuchs of UNC Chapel Hill. And Richard Marks of Sony on that panel. So you really, you definitely want to be there for that. We're also putting together an AR and VR contest, and we want you, and more specifically, we want your compelling AR/VR experiences, and we will be selecting the top 3 best experiences that you've created through the year and be showcasing those VR/AR experiences at the conference. That deadline is June 20th, so go ahead and take a look at— there, there it is, seagraph2014.seagraph.org. Or of course you can just Google that or talk to me and I can provide that information. There's my contact information and I'll open it to questions. Thank you very much. Do we have any questions in the audience? Yes. Hi. You talked about interaction and behaviors being important characteristics for virtual humans. You didn't talk at all about appearance. I'm interested to hear your views on appearance and, you know, the uncanny valley and all those kind of things. Yeah, so the uncanny valley, I was considering putting that in the slide. My belief is that realism, like photorealism, isn't that important. I believe what is more important is is motion. Like when you look down and you see some representation of your hands, you don't need the exact skin color. You don't, you know, you don't remember what the color of your shirt was. But if you just have something basic, even if the elbows aren't quite right, you're using inverse kinematics, it's really compelling to see some motion that is approximate. And, you know, some of the technical issues to virtual humans, such as latency, aren't as important as they are for the actual implementation for head tracking to, you know, have visual cues, you know, registered with the real world and such. So the requirements are different depending what you're trying to do. And a great example I mentioned was Sesame Street, right? Those aren't realistic characters, but you can very much relate to those characters. Cool. Thanks. Are there any other questions? Cool. Thank you very much, Jerald. Thank you. Or Jason, sorry. And thanks to the rest of the speakers. That concludes the sessions that are in this room for today.