Human-centered design for VR Training

Welcome everybody. This is Human-Centered Design for VR Training by Dr. Jason Jerald, CEO of Next Generation Interactions. We're talking about human-centered design for VR training. This is an ACM or Association of Computing Machinery talk. We're also doing this joint with Silicon Valley ACM SIGGRAPH. Is anybody from SIGGRAPH? Karl, do you want to speak to this?

Yes, I'm one of the officers for the local chapter here. SIGGRAPH is a special interest group on computer graphics and interactive techniques. And we try to hold roughly monthly meetings. We've been doing a couple of joint meetings here with, with the ACM chapter. We cover a variety of topics relating to graphics, movies, games, VR. Check out the Meetup, sign up for Meetup, and you'll be alerted to our next meetings.

Okay. Thank you very much, Karl. So a little bit about our ACM chapter. We're a local professional chapter. We were founded in 1957, so a little bit before meetup.com. We're focused on promoting knowledge of modern computing, want to create community, support networking and hiring. It's only a $20 annual membership. You can join on the website above, the sfbayacm.org. We list our upcoming talks on our Meetup site, which has about 11,000 Bay Area members who opt in. We have about 185 or 200 or so past talks on our YouTube channel. Talks like this one. We generally shoot for 2 monthly meetings, a general computing talk typically on the 3rd Wednesday of the month, and a data science SIG or special interest group typically on the 4th Monday of the month. And sometimes we do joint meetings with other things like SIGGRAPH, IEEE, or other societies. Education is not a zero-sum game. We all benefit. I wanted to speak about one of our volunteer opportunities. In the past, we've done judging at the science fair. There's a Santa Clara County Science and Engineering Fair held at the San Jose Convention Center. There's over 1,000 students from Santa Clara County. Our judges would sort out the computer science-related projects, judge them, and award cash prizes. This year in 2023, we were not judging, but we do welcome volunteers who might be interested in getting involved in the judging effort for next year. It's usually about March as a timeframe. Past student winners have used some advanced techniques. For example, drug discovery using AlphaFold neural networks and docking algorithms, cardiovascular disease diagnosis, an algorithm to determine eligibility for genetic screening of breast cancer. So if you'd like to encourage STEM study This is for both middle school and high school, and we also— part of our prizes are rewarding the teachers as well as the students. So we do have a call for volunteers. You can start with a limited engagement. Volunteers can join us monthly. There's different things that we can get involved in. There could be membership servers, services, help with online membership capability, publicity and marketing, member benefits. So for this meeting, if you're on the Zoom meeting, you can send messages via chat, Q&A in Zoom for questions to the speaker, if you have technical issues, and also if you have related computer science job announcements that your company's hiring for. If you want to post the, the job description URL, and if there's any contact information, you could engage some people to get interested in your company or your job opening that you might be hiring for. So now to introduce tonight's speaker. So the title is Human-Centered Design for VR Training. Dr. Jason Jerald is CEO of NextGen Interactions. So this is the first of 2 joint events on VR with Silicon Valley SIGGRAPH. When done well, VR training can result in more efficient workers, decreased injuries, and cost savings. This talk focuses on designing training results where VR can add the most value. The tutorial is for VR creators and stakeholders that are seeking insight in how to define, design, and evaluate impactful training. I'll go through a bit of the speaker's bio and he can add more details. Dr. Jerald is CEO of NextGen Interactions. He serves on multiple advisory boards and speaks at various events throughout the world. He's been creating VR systems and applications for over 20 years. He's been involved in over 70 VR-related projects across more than 40 organizations, including Valve, Oculus, Viturex, Sixth Sense, Google, Intel, AT&T, NASA, General Motors, etc. His work has been featured on ABC's Shark Tank, the Discovery Channel, in the New York Times, and the COVID of MIT Press journal presence. He's had various technical and leadership positions and has served on the ACM SIGGRAPH, IEEE Virtual Reality, and IEEE 3D User Interface committees. He's authored over 30 publications and patents directly related to VR, most notably the best-selling The VR Book: Human-Centered Design for Virtual Reality. So if we were in person, I would, I would invite the audience now to give a warm a round of applause for the speaker, but being virtual, we'll just change who's sharing screens.

Okay, thank you, everyone, for attending. I used to attend these events when I lived out in the Bay Area, so it's good to be here virtually. I wish I could be there physically. Okay, so yeah, if you're here, I assume you have some sort of interest or you're working in the VR space, and we certainly all know that VR is powerful. And it has the potential to make the world a better place. Here at NextGen Interactions, where I'm the CEO, we're a small team based, most of us out in North Carolina. We believe it's our moral responsibility to identify and work towards the most impactful use cases using this powerful technology. And so today I'll be talking about what I, and I think many others agree, is a pretty good use case of the metaverse and virtual reality and such. And that is training. Too many windows open here.

There we go.

Okay, so I'm going to start with sort of identifying some of the problems and the opportunities and also what to look for when you're identifying problems. Virtual reality can be used for quite a few different things, right? It's becoming quite feasible now. But instead of just what can we use it for, what should we be using it for where it can have the most impact? And then we'll move on to a couple other topics before closing. Okay. So I think we'd all agree that the world's certainly changing and changing at a faster and faster pace with these technologies, virtual reality, AI, all of those just moving faster and faster. And so these traditional methods are no longer enough or even realistic to continue doing the way things we've been doing them in the past. And last I heard, there were over 33 million people that have resigned as part of the Great Resignation, right? Not being happy with their jobs. In some cases, they have been happy with their jobs more recently, and there's been, you know, a lot of layoffs, unfortunately. Hopefully, people here in the audience haven't had to go through that. The World Economic Forum suggests that over 1 billion people will need to be reskilled by the year 2030. Now, you know, a lot of this sort of sounds like negative news, and the news is very biased towards negativity. But in reality, the world is becoming a better place, right? If you look at the statistics instead of just paying attention to the news, poverty, all those, you know, these things, it is becoming a better place. But that's because individuals and organizations are, you know, working hard at making that a world place, a better place. It just doesn't happen by accident. And I know, as you know, many of the engineers and such in the audience, you guys are working towards And so that's why I love talking with others that are up to big things. And so really, organizations as well as us as individuals, we need to reimagine how we're going to better prepare for the future and the workforce. One of my favorite quotes goes back— I think this was Alvin Toffler stated this back in the 1970s, but it's even more true nowadays than it was, you know, what was expected back then. And that quote is, the literate of the 21st century, which we've been here for, what, 23 years now, will not be those who cannot read and write, but those who cannot learn, unlearn, and relearn. What we learned yesterday may not be applicable. And so we have to, you know, always be relearning, not just teaching the younger generation. So we've all been in these classroom lectures, right? They're very passive. You're kind of You know, you're not fully engaged. As you can see, this guy, he is certainly not engaged here. Hopefully no one in the room was guilty of this. I've probably been guilty of that in a few of my classes. And the quality metrics, metrics typically with written tests, right? Or even if it's done nowadays on a computer, that's still often delayed, that maybe you're only taking that test where you get feedback you're not doing so well every couple weeks or so. That feedback is not immediate, and you're not getting that context of being able to actually experience how to apply that theory. That is, it's not necessarily applicable, that theory that you're learning. Or maybe it is applicable, but you don't understand how it's not applicable because you're not able to put it into practice. So we can take that one step further and we can, you know, do role-playing. If you had, you know, maybe some of the more engaging classes, if you had a good teacher, you'd actually act out some of those things Or you would have, you know, lab classes, right? But often that's very contrived. It's not really believable. Students, you know, we're not always the best actors. You're lacking the context even if you're in that lab environment. It's in a very limited, very specific problems. Now, in sort of the world that we've been working a lot with lately is with first responders. They— and I know, you know, the military and such does this as well— is live-action training, right? To go out and actually practice. practice these things, rehearse what things can go wrong, and practice those skills. That, however, can be pretty resource-intensive. In the public safety space, you might only have an opportunity to do this once a year or so because it can be so expensive. And so, you know, depending on the training, that can be $50,000 to $100,000 per day to do this training. And then if something goes wrong, you may not have an opportunity to, you know, practice that again until the next year. So you want to make sure you get right. And even then, the instructors often read from a piece of paper giving away the answer. And when you're practicing situational awareness, for example, then by reading from the paper, you're not practicing that situational awareness. It's not necessarily believable. They certainly have, you know, props and such, but you can't see the results of the successes and failures when you're doing things right or wrong. You're only really told of that, not— and that's not getting that full impact. So VR training is the answer, right? Well, not necessarily. By default, just because, you know, you throw a buzzword at something or, you know, you throw some budget or a technology at something, that's not necessarily going to get good results. And so with training, you know, bad training can be worse than no training. If you're training in the wrong way, then you might, you know, it's better not to do that training because you'll be doing things, you know, causing more problems than you're helping with. And the technology, of course, is only part of the solution, right? That's what we like to focus on as engineers, but really it's the everything that goes along with it. We really kind of look at training not as replacing traditional existing training, but figuring out where it's a good fit versus where, you know, things work just fine. There's not a need for that technology, but focusing on where we can really add that value. And most cases we looked at, we still need those quality instructors to be able to, you know, answer the tough questions, to be able to, you know, give more context of the specific situation or for the specifics of that location or the specific problem that they're training for versus, you know, generalized solutions. That could possibly change in the future, certainly with machine learning and such, but I think that's realistically probably quite a ways off. And then of course, all VR is not equivalent. It's not that some VR— and you know, you can just as easily replace this with XR being, you know, augmented reality or mixed reality, you know, pick your favorite acronym or term— but it's not that one is better. It really depends on what is the needs for the specific problem. And so if you throw a specific type of extended reality at a problem, that's not necessarily the right fit. So really understanding what is appropriate. But when you are able to offer, you know, effective training, then there can be that significant improvement of how people work and ultimately, you know, having those better outcomes. And so we're able to rapidly simulate the equipment, but also the processes, right? And then we can iterate upon those processes, try different things, take measures, and see what works, what doesn't work before maybe you even define what that training is actually going to be. And maybe that way, using these technologies, we can figure out how things to do better, not just replicate what's already done with the real world, and ultimately result in that better training transfer so that it has that higher fidelity of matching what they're going to experience in that real world under those time-sensitive operations. Ultimately, we'll have more accessible training. and lowers likelihood of mistakes and failures when things can go really wrong. So where is the VR? Where is the value for virtual reality training, right? Where should we be focusing our efforts? Well, a very interesting meta-analysis paper by Keplin published back in 2020, they concluded— I highly recommend to read this paper— they concluded that the value in XR as far as training goes is focusing on areas that exclude traditional methods that you just aren't able to do with traditional methods, such as situations specifically when danger or cost may make traditional training impossible. So that's really where we focus on because it seems that consistent with Kaplan et al.'s conclusion that that's where we can truly offer the most value and the most impact. And so to sort of summarize, in areas we look for here is when it's difficult to gain that real-world practice and experience and where that training is expensive and where the surrounding context is, you know, important. If you can practice in a classroom or out in a parking lot, then, you know, why go to all the trouble of creating these virtual reality environments in their training? And also where those behavioral and performance metrics are important. In some ways, it's a side effect of virtual reality. We're tracking the body in some ways, in some cases more than others. But we have that data that can be immensely valuable in being able to go back and analyze and figure out what people are doing. How are the experts behaving and doing things better than the entry-level personnel? What we found working with first responders specifically, pretty much the number 1 answer they tell us when we ask what are their greatest pain points are communication. In some ways, I guess that's not surprising. I mean, that's maybe common, you know, in other industries as well, but especially true with first responders when different agencies need to, you know, communicate to each other, when you have a call center trying to communicate with someone out in the field that's not seeing what they're seeing, those sorts of challenges. And then I also previously mentioned briefly situational awareness, to be able to practice seeing things that maybe you normally don't see because, you know, you're just not paying attention. And the chaos that can and confusion that can result in these bad situations. And also what we hear over and over again is for training in general, not just, you know, I'm saying general training, not just virtual reality training, is low-risk, high-frequency incidents are where the problems occur. And the reason for that is if you have these, these things that happen every day, that is they're high frequency, you're probably pretty good at them. You are practicing them every day. There's not much of a need to train those things. But if you only encounter that every 6 months or so, that's when things explode, right? That's when really bad things happen. That's when people die. So that's what you need to be focusing on is those high-risk situations that have that low frequency.

A question from the audience on the Q&A.

Absolutely.

From Jason, or no, to Jason. Thanks for taking the time for today's presentation. Good point. VR trainers are in demand. Is there any institution that trains the trainers for VR? Some sort of instructional design for VR?

Yeah, I know there's some companies out there that are doing it sort of on the developer or have, you know, to be honest, I haven't looked in a while. I know we were doing workshops with AT&T, but haven't done that in a couple of years. I know sometimes there are boot camps if you go to conferences. I don't know if SIGGRAPH has one. Maybe some of the SIGGRAPH people here know of some boot camps or tutorials. We occasionally do that. For example, we just submitted a tutorial Monday to the ISMAR, the International Symposium on Mixed and Augmented Reality, of some of the similar concepts here of focusing on identifying those high-risk impacts. That's not so much on the on the designing side in this specific case, but there are other tutorials and workshops at these conferences if you attend that you could participate in. Those are typically, you know, typically only a half day long or so, so they're pretty short. Obviously, universities are starting to, you know, offer classes in design more and more.

Okay, thank you.

Absolutely. Okay, so now as we kind of, you know, gave a high-level overview of the problems and opportunities, there's something called Kirkpatrick's 4 Levels of Training Evaluation that a lot of trainers focus on, right? And I think this just isn't important only from the perspective of training, but this has a lot of parallels with virtual reality design and identifying the problems we should be working towards, and, you know, the UX, the user experience, the user interface, how we iterate towards that. So it's interesting to kind of see the parallels here. Maybe this in some ways relates to the last question of, you know, how could we relate this to more of the design versus the identification of the problems? And so Kirkpatrick essentially divides this into 4 different stages, not just the evaluation of the training, but the development of the training that should be developed in parallel, evaluate as you go, right? A very iterative process. And the point here that Kirkpatrick tries to make over and over again is to start with the big problem, right? So start with the end in mind, right? What are you trying to solve? Forget about the implementation, forget about the technology you're using, but what are the pain points to start with? And then, okay, if we could solve for this, what would this mean, right? And so that's much easier said than done. In fact, most people that follow Kirkpatrick's 4 levels of training don't start with that, right? It's sort of obvious you do start with that, but people don't do it. They start with a level 1 reaction because that's the easy thing to do. Let's go and build something, and then let's get sort of, you know, end user feedback if they like it or not. But that doesn't tell us, are they going to truly be effective in changing that organization? And so that's a real challenge to try to define, okay, what specifically are those outcomes going to be, or what can we expect those outcomes to be, or what could be those leading indicators that tell us, you know, this is a good idea? The point is that you can spend a lot of, you know, money and effort of creating something that maybe there's not a need for. Now, you have to be careful of this because, as, you know, many developers know, you know, the waterfall methodology versus agile methodology. I'm not recommending the waterfall methodology, but I'm also saying you don't go 100% agile of having no goal at all towards— to work towards. You start with a goal, and then in the agile way, you work towards that goal, and that goal can certainly change, but you need at least, you know, attempt to be moving in the right direction. And so what is that intended impact that you're attempting to have upon the organization and the ultimate results to the world? And so what I'm going to do— and then these are lower levels, level 1 reaction being sort of people's opinions, right? Then the learning, you know, actually giving, you know, tests, for example, to see if their knowledge is there. And then is it affecting their behavior? Does that real-world behavior change? Okay, so it's in order to give some context for this, I'm going to use a case study we've worked on of kind of thinking about an example of Kirkpatrick's 4 levels. And so with hazardous materials specifically, this is essentially designed to train first responders, firefighters, which all have a basic— almost always have basic training in hazardous materials. But even though they have access to the equipment, and, uh, in this case, the air monitors to go measure for these toxic gases, they don't have the, the confidence to use this cuz they've never actually used it other than being shown, you know, a PowerPoint slide on a cla— in a classroom setting. And maybe they bring the device on, you can turn it on, but you can't actually see what, you know, happens when you put it in a toxic gas. Because to do that, you need a toxic gas. And generally it's not good to release, uh, toxic gases into the environment that, you know, if you breathe it in, it's a problem. And so starting off, what are the pain points of hazmat? Well, the, you know, to start off with, the most important thing is the human element, right? Thousands of people, hundreds of thousands of people are injured or killed annually due to hazmat-related incidents. You often don't hear about these, but sometimes you do. You know, sometimes you hear about a train derailment. or others that, you know, makes a big deal in the news. But it's actually happening much more often than, um, than we see in the news. And working directly with first responders, we hear these horrible stories, whether it's with just hazmat or, you know, general first response, is really horrible stories. And so, uh, an example of that is, uh, there was a call where a family had reported some, uh, an odor in a, in a household. So the team had come out and they used these air monitors, um, in order to— essentially, you take these air monitors, you move them through the environment, and you hold them for— you need to hold it in order to get good readings in different areas. Some gases are higher and, you know, are lighter than air, so they raise. Some are heavier, so they congregate. They have different behaviors. But, um, unless you've done this a lot and you've been in situations where it often occurs, you don't necessarily, um, able to detect those gases. So in this, uh, the one example story. After the, um, the hazmat operations were called out to the house, they went through the house, they took measurements, and, um, no gases present, right? So they gave it the all clear and, um, got the family back in the house. And, uh, then the next, uh, next morning, the, the parents had woken up, uh, they were fine. However, um, the kids were not. The kids had Never woke up, right? 2 fatalities, 2, uh, 2 younger girls. And the reason why they had reported the odor of the gas, no one else smelled that gas, is because they were really young and down towards the floor. And I believe in this case it was hydrogen cyanide that was very heavy down towards the floor. And, you know, different cross-sensitivities and such on the meter that the, um, the operators out there just weren't able to detect it, right? So can you imagine? I mean, obviously for the family how horrible that would be, but also the first responders of, you know, how bad you're going to feel if something like that happens, right? So this has a real human impact that, you know, can be life or death in these situations, and plenty of other horrible stories. Then of course the environmental impact, right, on these larger type of events, not just explosions, but, you know, things getting in the water. I know my My mother, for example, with a chemical leak in Tucson, Arizona back in the '60s, she had major internal issues her entire life. And I have cousins that have these problems because of that, that had that environmental impact of those chemicals released into the environment. And of course, that can result in economic impact, right? If you want to put numbers to it. So hundreds of billions of dollars, due to, you know, displacing families and communities and businesses that, you know, often clear out areas and such. So pretty big pain points. And so when you hear these stories, I mean, it kind of gets to the point, yeah, we can use, you know, technology to try to help people. But when you hear things like this, and if there's any chance of being able to help them, it becomes like, why would you not want to help them? All right, this is something that We should or we must, you know, if we have the capability of helping them, then, you know, we really should be. And so to be able to practice as if they're actual incidents, right, you can use virtual reality to do that. These actual incidents that are truly experiential, right, outside of VR, I'm saying when they actually occur, they rarely occur, but when they do occur, you're going to remember them for life. Right? When people die, you're never going to forget those lessons. However, because they're so rare, it can take years to gain competence, right? If you might only have, you know, say 10 of those, a handful of these, you know, critical events in your entire career. So how are you supposed to get good at it? How are you supposed to get, you know, confident in those situations when these horrible things are happening? I do have a video here. Let's see. I was having problems, so I was just going to see if I could share on YouTube here an example. A lot of times you can't capture this, right? But this is an example of something going wrong, uh, out here on the East Coast and, uh, Maryland. Got a camera. Okay, we typically don't think of hazardous materials as exploding, but they can, right? Very different than the household, you know, setting that I just, I just went through. But bad things can happen when you have these chemicals in tight spaces and such. Okay, so that's the beginning of defining what are the pain points. Are you looking at the right problems with Kirkpatrick's 4 levels, right? So the 4 levels of results. Well, we, you know, trying to get a good idea of, okay, how are we impacting the organization? Well, these training costs I mentioned, these live-action trainings can be $50,000 to $100,000 for a day of training. And if you, you know, you miss it, you got to wait until the next year, you got to go to another state sort of thing, right? So this is obviously, you know, there's an opportunity to decrease training costs there. But more importantly, increasing that overall preparedness for when things do go wrong. Hopefully they don't, but if they do, you're going to be able to be prepared to properly handle them. And so through proper detection of these hazardous materials, we literally have the opportunity to help save lives, right, by better training the workers in this area. And of course, like we talked about, you know, being able to mitigate, you know, certainly we're not going to get 100% of the wear. way there, but to reduce that economic impact that they're having on our families and communities and that environmental damage. Okay, so the next level is behavior, and there's kind of a couple ways to think about this. One is, you know, true behavior, not just in the training, but after the training. Are they actually behaving different in their jobs? Now, a sort of advantage of virtual reality is that you can put that in that environment and evaluate them in the setting that might be different than the training, right? You can put them in sort of a different scenario and get an idea, okay, now are they doing what they were trained for? So here is an example of just the behavior of some firefighters here in Raleigh, North Carolina, the Raleigh Fire Department, of using the system. Right? Being in that environment, going in as a team and practicing the use of those meters, but also being able to be evaluated with an instructor in there with them of how are they doing the proper behavior. Specifically, the most common need we find, which is, you know, surprisingly simple, but just hazmat workers aren't doing it even though they know they're supposed to do it. You sort of, you know, have that knowledge. Yeah, of course I'm going to be doing that, but you rarely do it because they're not practicing it, right? Is that holding that meter high? medium, and low, all, you know, all the way near the ground. We had one, uh, hazmat, um, hazmat worker, for example, um, that came in and had used our system. And at one point he said, wow, I just realized that, um, I've always been doing it wrong. So we asked him, you know, well, what do you mean by that? And he said, well, I just realized that, you know, I've always been holding this air monitor By low at the knee. But now that I can see the gases through virtual reality, now that I can see the gases are actually settling really close to the floor, and my meter, my simulated meter, is not reacting— I'm not getting much of a gas reading at all when I hold it on my knees. I have to hold it near the floor where I can actually see the gas, right? Because in virtual reality you can see the gas that you can't see in the real world.

Mm-hmm.

And says, now I realize what low means. It means on the floor, not at the knee. And so I said that, that was wow, right? So I asked him, uh, how long have you been doing hazmat? And he said about, about 10 years. So he'd been holding low near his knee for 10 years, right? So what if he went in that household where the kids detected the odors and didn't, you know, actually hold it near the ground and hold it there for enough time for that sensor to get that reading, right? But here in a few minutes, he was able to get that concept. So I thought that was a really good example of where, you know, insight, real true insight and behavior, now holding it closer to the floor versus the knees, how that, you know, he's going to start doing that now because he's going to remember that. And then if it's too much of a problem, it's beyond your capabilities, knowing when to call in the specialist. Sometimes a better behavior is no behavior. Get the heck out, especially in this hazardous material sort of situation. So learning, right? Here on the right, we have some examples of after-action reviews that you can go through and look at and evaluate. Okay, how were they actually performing? And so this is kind of midway between behavior and learning. But the point is, with the after-action reviews, you can go back and learn what you did right, what you did wrong. Did you miss a gas? You can see here is turned on some of the gas visualization of where I detected the gas over here, but I completely missed that one, the gas leak over in the corner and then also talk about it either within VR or outside of VR of what went well, what did not go well. And the nice thing about the after-action reviews is there's no argument of, yeah, I did that. Okay, well, let's go back and replay it. And instead of arguing about it, let's just look at the data. And then reaction is what a lot of us do is as developers, or especially UX designers, right? So if there's any UX people here, you probably know about, you know, the System Usability Scale and different forms of evaluations. If you're on the marketing side, maybe you know about Net Promoter Scores, right? And be able to track— we track that over time, right? Okay, hopefully our System Usability Scale is going up and not down as we continue to develop things. And a lot of that is not adding features, it's simplifying features and removing things that aren't necessary. So that can be pretty important. But the point is, do this after you've done the other things, right? Work backwards. Start with the end in mind, and then work towards this instead of starting with, we're going to make the most usable system ever. Well, so what if you're focusing on the wrong problem? Okay, so to conclude, sort of as a result of, you know, working closely with subject matter experts and such, we have an example video of our training here. By the way, do you guys hear the audio in that last video?

Yes, we did.

Okay. So if it's too loud, I'll see if I have control over my volume here.

No audio at this point.

Is that too loud?

No, no audio.

No audio. Okay. Well, I will talk over it. Okay. So You know, speaking of usability, people that have never done VR before, we're starting off in a basic environment in order to get comfortable before going into the actual scenario. And also the instructor is able to, you know, kind of explain the scenario they're going into. Now, hazmat workers always go into pairs. Here we have 3 users because the 3rd user in this case is the instructor. So he goes in with the students and can explain things as you— as they go. And then if you're going to evaluate them as a 2-person team coming later, than, you know, just 2 of them can go in by themselves. So pretty simple concept. You have a device that detects these simulated gases, and you go in there and you practice detecting those gases under different types of scenarios. And it's more than just, you know, detecting the gas. There's actually a lot of theory, right? This is chemistry that, you know, how the sensors have cross-sensitivities, even if it's not directly detecting the gases, you know, Kind of unexpected things can happen. Here you can see just the gas simulation that, you know, maybe some of you here are doing computational fluid dynamics of, you know, just simulating that. Then the sensors are the simulation of detecting those gases. Here's just a first-person view. So we have a desktop view. All of this is taken from the desktop view so that you can, you know, bring up data from different perspectives. Okay, so I don't think the, you know, the point here isn't, oh wow, what a cool demo. It's more, okay, how does this apply for your actual target audience? Right? To be honest, people that don't know hazmat, that's just, it's not easy to see the value here because if you don't understand the problem, it just seems kind of like another VR demo, right? Which we're proud of. You know, our team has done an amazing job. But, you know, there's a lot of really great applications out there. But are you focused on the right problem is key. Okay. And then kind of, okay, what are those benefits that your solution provides over, you know, what's done in the past, right? Whether that's classroom training, live-action training, or even on-the-job training. Even on-the-job training, those are, you know, actually being in an incident that you know, rarely occurs, you still not— may not be getting the insight if you're not able to actually see the gas. Like, someone may die and you don't know why, right? So with proper training, you can actually learn more than you can of, you know, rather than actually being in the incident. And all the other sort of things that are, you know, valuable with VR, right? 24/7 access versus, you know, once a year sort of, you know, live-action training. Okay, so now that we had an example of how to think about Kirkpatrick's 4 levels of training, right, not just in the evaluation, but really that helps to define and design what you're creating. But now moving more towards, you know, lower-level specifics of human-centered design, we'll go into some of those concepts. And there's a lot of concepts, just, you know, some examples here listed, kind of discussed in my book. We certainly don't have time to go through all of these. But a couple of these I thought we'd go through and have some fun with. So there's all these sort of different tools out there, virtual reality being one of those, to be able to simulate and figure out what works, does not work, is a great use case for virtual reality as well. But one of the most exciting virtual reality techs I've ever seen blows all the others away, and it's the most exciting. And I'd say it's absolutely essential with any type of virtual reality, augmented reality, augmented reality, or really any technology, is, is not really possible without this specific thing. Many people are only vaguely aware of this, or maybe we take it for granted. We're not always aware of this, and it's been around for, in various forms, for thousands, I'd say even millions of years in different forms. So in the chat, I'm curious if anyone— let me bring up my chat here so I can see the responses. If there's anyone that knows, uh, knows what I'm thinking here.

There was a comment in the question and answer. Monica was asking, does the target audience, you know, company internal training, uh, seek you out, or are there startup founders who found their niche and want your company to flesh out their idea?

Oh, that— so ideally, yeah, the founders, right? I know there's some great, uh, companies out there in the first response side, for example, that former police officers are the founders, and that's pretty ideal. And so, yeah, I'd say that's the best case. That's not always the case because you don't necessarily— when you found— when you create your company, you don't necessarily know what you're going to be working on, especially if you've been around a while and you take on different projects and such. So, yeah, I'm specifically thinking, who are your subject matter experts? We hire subject matter experts, right? So we hire part-time first responders, right? We get firefighter and hazmat technicians. Or hazmat instructors on our team. But yeah, if you can get a founder, then yeah, that's even better. I certainly don't have, you know, a background in hazardous materials, but I'm learning a ton. It's fascinating. Okay, so I'm seeing some, some, so stereoscopic vision, storytelling, human imagination. So I play, so I'd say all of these actually are subsets.

One question is, are you involved in, like, some of the video game engines for generating some of the graphic images, like Unreal Engine or one of the others? Do you have particular favorites? I've worked with some generating images for image recognition models just to have synthetic data.

Yeah, we really, like everyone, I guess, we've really started excited about machine learning. And are starting to explore that more on the augmented reality side, what that might mean. But as far as what engines we use, we used Unity, and Unreal Engine is great as well. But, you know, Epic Games, their headquarters is a mile down the road, so they hire all the great, you know, Unreal Engine developers. So we stick with Unity. Okay, so the answer to this question, the most virtual— exciting virtual reality tech, and none of the other, you know, it's not— I'd say by definition it's not virtual reality if you don't have this technology. So it's kind of a trick question, and I think you guys had it right because, wow, this is also required for, you know, storytelling and stereoscopic vision, I'd say, is certainly a part of this. And it's also required for play. I guess it's It's necessary for anything as a human, no matter what you're doing, because it's the human brain, right? So a massive number of neurons, which are the basic unit of the nervous system. But what's really more impressive than the neurons by themselves is the connection between those neurons, right? There's on average thousands of connections or synapses from a single neuron connecting to other neurons. So that results in billions of these synapse connections For just a cubic millimeter of brain matter, right? It's, it's mind-boggling to think about that. And this, when I saw this and over and over again, I said, that can't be right. And I, you know, I mistyped that and I'd go and look again. But sure enough, every time I go look, 150,000 kilometers of nerve fibers, right? Enough to circle the Earth multiple times. That Every time I say that, I like pause because I'm like, that, that's just not possible. Then I go look at, you know, I go look at the source and indeed that's the case. And so a neuron can reach any other neuron on average by traveling through only 6 other neurons, uh, resulting in this mat— you know, we talk about parallel computing now. Well, this is massive parallel computing inside our heads and we just, you know, happen to call it a brain instead of a computer. And so it's really, um, when it's combined with the body, it's a perception and action entity that enables is to interact with the real world and also the virtual worlds, right? And how you act with the virtual world, if done well, that's how you're going to act in the real world, right? Our brains are fooled. And so today's computers, I'd say, cannot compare. We'll see if that continues with, uh, with AI taking off the way it is. But, you know, some humans are good right now anyways at some things that are better in humans and vice versa. But really, it's comparing apples to oranges. And so, uh, when people say, you know, when is it going to Is machine learning going, or AI going to surpass humans? I don't know. That's a tough question, because how do you define surpass, right? There's just so many different factors there and ways to think about it. So here's a version of Goldstein's iterative perception-action process, and this is really a model of how distal stimuli out there in the world reaches our sensory receptors, and then how our brain you know, processes it and we act upon it with our body. And so the distal stimuli out there in the world, we pay attention to it. Thus, we kind of orient ourselves towards that, you know, move our eyes towards it, for example. And then that's transduced into signals that the brain processes. And then we perceive, magically perceive those distal stimuli through all these, you know, massive number of neurons and synapses. And we recognize objects at a higher And then we can act upon the world and modify that distal stimuli, and the process continues in this big circle. And this can really be simplified to simulate— I'm sorry, be simplified to the stimulus, right, being that stimuli out there in the world, to the physiology of our brain and our body, to perception, but then back to stimulus as we reach out and we interact, whether it's the real world or the virtual world.

world.

However, this is what's more important, and part of my point here is this more than just this, you know, automated stimulus-response iterative cycle. That is, we're more than just a function of the environment, right? This top-down knowledge such as our previous experiences, our memory, beliefs, you know, what are our goals we're trying to achieve, that affects how we perceive and act. And I mean, that's just, you know, standard psychology that you— many, many examples of that, of You know, you see what you pay attention to, as an example. But that allows us, by thinking at a top-down level, you know, as humans, that's what makes us really powerful and maybe unique in some ways to other animals. It allows us to tweak the cycle and the world in a more, you know, conscious manner versus just an immediate results-oriented manner. And so for virtual reality, all we do is we put in a VR system between that action and that distal stimuli, and that becomes our XR input, right? How we reach out in the world, how we, you know, click on something, how we look at something, if something's going to respond based on what we're doing.

A question from the audience, if this is a good time to bring it up.

Yeah, yeah, please.

How much time does it take to modify your exercise? To adapt them to different hazards? For example, a flammable solid with a low melting point that reacts with water.

Yeah. So, boy, different ways to answer that. So as far as gases, that's pretty straightforward because that's just defined by, like, you know, the molecular weight and such, right? Do they rise? Do they fall? Do they have different behaviors? And if we've implemented those, you know, similar behaviors, it's a configuration change. Now, if it's something more specific, of course, we're going to have to, you know, one, understand the problem and then implement it. In our case, and I suggest this for anyone developing systems, well, maybe not as a first pass. As a first pass, you know, as one of my advisors in graduate school, he said, you know, build one to throw it away, right? But after you started building a few of these, start architecting it so that you can reuse components. So for example, our, you know, and this is, you know, pretty standard for for software engineering, at least at the architectural level, of you want to have that reuse, right? So for example, our after-action— some of our tools that are common for training irrespective of the type of training. So for example, we found, you know, talking with, you know, not just hazmat people but other first responders and other people doing training, after-action reviews are pretty important if you want to evaluate and discuss what went well, what did not go well. And so those after-action reviews are those common tools at a higher level that are sort of part of our development platform that, you know, our hazardous material training just happens to use. So in some cases we can reuse that, that code. In other cases, yeah, we have to rethink the problem because it can be quite a bit different or, you know, more specific. So someone mentioned, uh, seeing the circular diagram. Um, are you seeing more— like, are you seeing more now or only a circle by itself? There was a fellow who was seeing the video, the YouTube video still, I believe, that you had up earlier.

Oh no.

And he has now— I've corresponded with him a little bit in chat, and he's now going to try re-logging Yes, and he says now it's fixed.

Okay.

Okay. And I see, Greg, you see the circular diagram. Are you seeing the human action equals VR input tracking on the right-hand side?

Exactly. Yeah. Okay. You're fine. We just didn't want to bother you while you're presenting. We're just handling a technical issue on the side. So.

Okay, great. Okay. Glad you got that worked out. Okay. So, you know, with VR, all we're doing is— well, I guess more than all we're doing because it's, you know, a lot to it. do it, but we are hijacking the human perception-action system and replacing that real-world distal stimuli with the artificially created cues that my— the mind interprets as a form of reality. And if this is done really well, which I'd argue we're not there done, we're far from that in many, many ways, but in other ways, right, you don't have to have this perfect reality that you're trying to replicate. But if you're focusing on what are the most important things Instead of trying to get everything perfect, right? If you're trying to, you know, train for muscle memory in a very specific way, then that's what you should be focusing on and not making, you know, the visuals perfect, right? So maybe you should be focusing on the haptics and then the visuals can be lower quality. And so essentially in virtual reality, even if they're cartoon worlds, right, then you can experience that sense of presence in that artificial world, even if it's not And then, you know, hopefully that moves over when they go into the real world, if it's designed really well. Now, the point here was that top-down knowledge, right, is that we don't perceive with our hands, our ears, our hands. We perceive with our brains, and our perception of the real or the virtual world— a world, I guess, whatever it is— is not reality of what the truth is. But it's an interpretation of that reality, right? So if I, um, you know, whatever I have on my desk, I have this phone here, right? This is a phone because we call it a phone. Or I have this, you know, chair sitting over here. That's only a chair because we agreed to, you know, call it a chair. It's just a mass of, you know, a mass of atoms, a lot of atoms in a certain, you know, shape and certain materials put together. But the truth is they're just, you know, a mass of material put together. And then we're not perceiving it directly, right? Those are those photons bouncing around and entering our eyes, and then we perceive it as a chair and we agree on the chair and such. But the point is that sense of reality can be very distorted and different for different people and under different circumstances. So, you know, the joke here is there's an old man, this is how he perceives himself, right? Uh, maybe this is myself, I don't know. But, uh, the point is, you know, we might perceive this guy differently than he perceives himself. Or maybe a non-visual example is a song can be quite emotional and have some significant meaning for one person while not for another person, right? If I, you know, broke up with a girlfriend in high school that, you know, I was, you know, crying over, might not mean anything to you, but maybe, you know, maybe I start, you know, tears come to my eye, you know, myself. So the point is we need to take, understand, and work with the end users to really understand how they perceive the world, how they act on the world, and how they think about the world, getting inside their minds essentially. And so you need to work with these subject matter experts. As is one of the— someone had asked about being a founder, even better if you're a founder, right? Because you're really going to understand that, the problem. And that's the power if you're a company to focus on something niche, something you know well. At the same time, there's always trade-offs being able to be open to with, you know, ideas outside of, you know, listening to other people's problems versus being hyperfocused on something, not seeing stuff outside of that. Yeah. So, I see one of the questions on distal stimuli and perception. Oh, I see, Greg, you're sort of, oh, you gave the answer. So yeah, probably a better answer, a more official answer from apa.org. So yeah, distal stimuli are something that's just Out there, right, in the world. Then proximal stimuli—I mean things that are actually touching the senses, right? There, the photons are falling on the retina of your eye. On an organ is on a sense organ. Okay, and so yeah, here on the slides, working with the SMEs, be a SME yourself, or you know work on gaining that knowledge if you're not, you know, understand it yet. Take classes on the subject, right? Gain that deep understanding. The great thing about working with SMEs is they'll introduce you to other SMEs that maybe, you know, have a slightly different perspective, or, you know, or you are doing things in a slightly different way. So you don't want to just talk to one. You want to talk to lots and lots of people, which we all know from, you know, agile development, or, you know, getting out there in the world, right? Talk, talk with your target audience. And I would go, you know, also say You know, don't ask your SMEs about how would you use virtual reality technology. You'll get crazy responses because they, you know, that's not their job. They don't think about the technology all the time. They may not even know what it is. You may be able to explain it to them or give them a demo, then maybe they'll have ideas, but that's not their area of expertise. So I think, you know, starting with the pain points, not the technology, and not jumping to the solutions, but really understanding You know, their pain points. And it may be that virtual reality or your, you know, your solution may not be a fit. And if it's not a fit, that's really good to know. You know, move on to something that is a good fit instead of trying to force fit it and, you know, using it for something that's not— that isn't a fit.

A question: working with subject matter experts, do you ever get the the effect of if you have one watch, you know what time it is. If you have 2 watches, you never know what time it is.

Oh, yes.

Where there's a lot of minor conflicts and things that don't quite jive. They have good experience. It's just have different histories. And so they have different perceptions when it gets down to, you know, splitting hairs on a given situation.

Absolutely. So I'd say, you know, it was really frustrating, sort of because, you know, I'm not a patient guy. I want an answer, you know, yesterday sort of thing. And so trying to figure out, you know, not just the technology and how to be used, but the business side of things has been a fascinating journey of— we probably had to start— we probably talked to 100 first responders until we started seeing, you know, trends, right? It's like everyone would have a different answer every time we asked a question. And so, you know, even when we went to 10 of those, because, you know, they were broad, you know, general questions, And I'd say we had worked on that for 3 years. We received some government funding for 3, 4 years of support before we kind of determined that hazardous material training was a really good fit. We kind of had a pretty good idea it was a good fit, but there were other, you know, factors that came into play. So we looked at a lot of different use cases, and everyone had a different opinion, and some had like Like really good points. For example, character behavior, how realistic does it need to be, right? One of our subjects, you know, some are like, oh, it's good enough, right? It's just if it looks like a cartoon, it's fine. Others like, you know, if someone gives a slight twitch or, you know, really subtle body language, right, can be really challenging to do, to simulate in VR, right? Because we hardly understand that in the real world. So how do we create it in VR? I think there's a lot of opportunity on the AI side of, you know, reading real human behavior, then, you know, replicating that even if we don't understand it perfectly, maybe there's, I think there's a whole lot of potential there. But the point is, yeah, it would be different answers depending who you talk to, even though, you know, they have the same job description. And so we had to talk to a lot of people. But after we started noticing those trends, then we started noticing it when we did have a hypothesis, is this important? We'd start getting consistent Oh yeah, that's, that's definitely important. But we didn't get to that right away for sure. So that can come. That's a great question. That's very related to the slide here, right? Well, that's a lot of one-on-one discussions, having focus groups, you know, truly experiencing a police ride-along. We had one of our guys do a ride-along and it happened to be that case. There was a really bad incident that he was in the He was in the police vehicle that was the first to scene and was in the news and was a big controversial situation, and he witnessed it firsthand, not through those cameras. So it was a pretty crazy experience for him to, you know, safely view it from the seat of a police vehicle. Questionnaires. I think you have to be careful of questionnaires. When we have sort of general questions, we get general responses. And so You really should be doing those focus groups to get to the right questions and talking with subject matter experts of, you know, asking the right, you know, specific questions, what you're trying to get to versus general questions. And then I previously mentioned some of these other things from the UX side of things, doing these formative usability evaluations, you know, talk-out-loud protocols and such. And the System Usability Scale is— I don't think it makes sense to say, you know, So overall, that, you know, absolute scores, especially with virtual reality, sometimes people get really excited about it, so they give higher scores. But, you know, we've gone from, you know, when we started having a SUS in, you know, the 70 range up to, you know, a 90 range now that we've, you know, dramatically simplified things as much as possible. And, you know, you just put on the headset and it works versus having to figure out, okay, what, where's the button, you know, to do Do some task, and that promoter score you can also track and you know over time as well. Okay, so another example is you're talking to talking to subject matter experts and your your audience. Sorry if I said SME there. Yes, SME is subject matter expert. You know whatever in that could be you know subject matter expert in engineering on the military side. You know maybe an artist whoever that might be someone that really knows their. So understanding the constraints, right, of how the, uh, how these workers operate in the real world. A lot of times you see something like, why the heck do you do it that way? Why don't you just do it this way? It seems so obvious. Or, you know, and, you know, when I watch, you know, a good film, you know, that firefighters do it this way in the movies, right? Why don't you guys do that? Well, usually, you know, getting your information from Hollywood films is not where you're getting the true information.

Yeah.

So, you know, There's good reasons why they do and do not do things certain ways, and part of that is constraints. So Dr. Fred Brooks, my graduate school advisor, he has a book called The Design of Design, which is a really amazing book, in addition to his some other books he has. But he talks about constraints. Constraints. Okay, what does that mean? Right? Well, he breaks it down further than just constraints. Just, you know, constraints, sort of, you know, the general thing. What does it stop you from, you know, doing some things, right? What is holding you back? Well, he breaks it down to these different types of constraints. So real constraints are true impediments that cannot be changed, right? They're sort of the truth. And then resource constraints are real-world limitations of supply. So budget, right, is a good example of a resource constraint. Maybe, you know, time is another resource constraint. Obsolete constraints are— were once real constraints, but now they're no longer valid, right? What was true yesterday isn't necessarily true today. Misperceived constraints, people think are real, but they're actually not, right? So a wall in virtual reality might be a misperceived constraint. You can stick your head through that wall. Indirect constraints are side effects of real constraints based on assumptions or misassumptions of how something is achieved. And then intentional artificial constraints can be useful to buy for design in order to narrow the design space and enhance the experience, right? They can be beneficial. So constraints don't have to be bad. Okay, so now I want to jump into— and I just realized it's about 11 o'clock, so I'll go through these quickly. But I also want to make this interactive. So please, if you know the answer to this— some of you've seen this puzzle before. So if you've done the puzzle before, that's okay if you give the answer. And I don't know if you could how easy it'd be to type the answer, because obviously it's easier to draw. But in this puzzle, maybe you've seen this before, can you draw over all of the dots with 4 straight lines connected end to end? And as an example, you might do something like this, right, as a starting point. So I'll give you guys a moment to maybe, if you have a notebook and pen, To you know, just think about it in your mind. What would be the next step? Okay. So are there constraints here? Right. This relates to this previous slide on constraints. What are the real constraints? Of course. The clue is think outside the box. Be creative. This is more direct thinking outside the box versus, you know, when we generally say think outside the box. Right? That box is not a real constraint. That is a misperceived constraint. You just take your line, go outside the box, and now you can come up with a solution. So simple example of a box being your constraint. Okay, so next, next question to make this a little bit more difficult. Ah, 3 lines are sufficient. So, um, maybe that's this question. So Carl maybe has an answer. So Carl, um, what are your— how would you do 3 lines where they're connected end to end? I suspect he is typing. I'll give you guys a clue as he's typing to think even further outside the box. So who said the lines had to be— there you go, right? Who said lines had to be, uh, horizontal? Good, Carl. Most people don't get that.

I can do one line if it's 100 pixels wide.

There you go. That's a good example. So you are thinking beyond the box. But yeah, what is the definition, right? What are your assumptions, right? One, people may think that a line, you know, what is that definition of a line? And there's another answer. So that's one good answer. And there's another— well, here, so Yeah, now draw over all 9 dots with 3 straight lines, but this time the lines cannot extend beyond the box boundaries. Okay, so let's go back to our list of constraints. What about obsolete constraints? What was true in these other questions is no longer true. This does not state end to end. They have to touch end to end. It's really easy to, you know, just take the same question and reuse that question. This is a different question. It doesn't say they have to be connected end to end like it does there. So once you do that, it's just 3 lines. Okay, and Greg, you jumped ahead. Thank you. So we can go through this one quickly. A single straight line. You were ahead of me. So we could answer that first question with, with this answer. You know, use 4 lines, but we only need one. Okay, so yeah, quickly go through these, what people think innovation looks Like, right? We put in some effort and more effort, more results. Then you're happy. I call that the fantasy method. And then you know enlightened people. We think about well yeah, you work really hard at something. Maybe you're not getting results, but if you keep pushing, right, then you get some results and then you're happy. But you may not see that initially. And the joke here is I'm going to go through these pretty quickly. Is this is how it really looks. Now this graph doesn't even make any sense. any sense. It's supposed to be a joke here that there's a lot of ups and downs, that there's, you know, sometimes you feel like you're going backwards, all of that. And so we call that the Zigzag Method, which relates to, you know, levels of competence. You think you don't know what you're— you don't know what you don't know. And then once you know that you don't know it, you're not very— you feel incompetent, but you're actually more knowledgeable than you were before. And that's a big iterative cycle, right? Then you realize, okay, I'm starting to get better at And then you get so good at it that you don't even realize you do it. You just kind of, you know, do it automatically. That's one of the challenges of learning from experts. It's just they don't even think of how they do the job. So that's one of the challenges of working with, with subject matter experts. So you really got to get inside their mind. And sometimes, you know, it makes sense to talk to the newer guy on the job, right? What is his frustrations? It's maybe not a frustration of the more senior guy. And so Thomas Edison said, I have not failed. I just failed 10,000 ways going through these very, very quickly. And how to fail 10,000 times fast is become an XR developer is kind of the joke. But the point is here with training, there's a lot of corollaries between how you're developing these VR experiences to how training works, right? So I kind of think about both and how do they relate. And so with training, right, if we create these systems, we can allow a lot of failure and to be able to learn from So iteration, right? You define it, then you make it, then you learn from it, and you go do it again. Of course, you guys are all— I'm sure, you know, this for being, you know, in the 21st century, we know agile development. So this is how we sort of, you know, work with customers here at NextGen. I suggest you guys, you know, maybe have similar processes in place or put together processes with your own customers. So this is how we do it. Right, of really, you know, starting with that high level of, you know, what are the pain points that, you know, Kirkpatrick's Level 4, what are you trying to solve for, and what are the expected outcomes? And then determining if VR is a fit. And if it's not a fit, then that's really good to know. That's better than putting a bunch of time and effort into it, then realizing you wasted all that effort, time, and budget. And so you got to be able to tell, you know, customers, no, you know, I'm sorry, this, you know, this isn't worth working with us on. Maybe they have other problems, maybe they don't. But you really want to focus on where you can have the most— add the most value and finding the right customers that can benefit from the technology. And then if it is a good fit to work with them in collaboration with them, not for them, you know, not talk to them at the beginning But throughout everything, work with them and iterate with them. And then pilot it, you know, once you think you have a pretty good solution. Again, part of the iterative cycle, confirm that it's going to work before deploying to a larger workforce. Okay, so very quickly, maybe I'll skip some of the other slides and end on this part of the conclusion so that we can move on to questions. I want to make sure we have time for that. If, what if none of us feared failure? Well, with virtual reality, we can fail in a safe way, right? And not have to fear if we're doing something wrong, right? And hopefully reflect upon it and learn from the experience, not just, you know, have fun from the experience. And especially if there can be metrics in place, if there can be that feedback in place, then you learn from that failure, which in some ways can be better than learning from your successes. Obviously, you want to learn from And so we have the opportunity to create and experience new worlds, whether it's, you know, for training or other things that will have impact and really make a difference in, you know, the real world, not just, you know, in these virtual worlds. Okay, I think we'll skip some of these so that we can move on to questions and answers. Oh, I also— someone is asking about the code. So here's a discount code for my book if you're interested. You can read more about it at thevrbook.net. And if you like it, please consider giving a review on Amazon. So that's SFA25.

Thank you. There is a question from the audience from Monica. Does your question pay for your problem discovery process? Or is it part of the package deal with the end result training program?

Yeah, so typically, you know, it starts with that discussion, but that being part of the most important aspect, it really— so a lot of times, for example, we will co-propose with an organization. So for example, if we're applying for a government grant, we're not going to charge for that sort of thing. But if it comes from, you know, an industry customer, then yes, that would include sort of that. We've done a lot of work of— right, because we see the value of identifying where it is not a fit as much value of where it is a fit. And so that could be as valuable as, you know, actually doing the work, as saying, I don't know, I don't think that's probably a good fit. We don't want to be biased towards we only get paid if we find a good solution. Are a good fit.

One question, when working with subject matter experts, sometimes it's good to ask about their different types of bias that they may not realize, like if they have a contextual bias or cognitive bias, you're trying to interview them, unconscious or implicit bias, just things like that sometimes are helpful.

Oh, absolutely. So Well, I don't know if there's a good answer for that because as humans we all have bias. But I think one is talking to a lot of SMEs and then also a, you know, diverse set of SMEs, right? You know, I guess firefighters are diverse, but they're not that diverse compared to, you know, other industries. So that's actually—

Sure.

A big challenge for us as far as, you know, their backgrounds and such. And although you'd be surprised, I've met like volunteer firefighters that have PhDs, right? And so I have a friend, a good friend I went to high school with, one of the smartest people I've ever known, got his PhD in chemistry. And he's a volunteer, you know, he's a retired volunteer firefighter. And, you know, you know, understands hazmat, right? And so you'd be surprised, there is some diversity there, but certainly not as much as other industries. So we have to be challenged with that. Outside of talking with SMEs, I think virtual reality We're looking at doing a proposal now for bias training, right? Because now you can actually, you know, keep everything identical, then change one thing, like, you know, change the color of someone's skin and then measure and see if there's a difference. And, you know, then there's no argument. It's not someone's opinion that, you know, someone had bias or not. It's like, sorry, we only changed one thing and people, you know, behave differently. And so I think there's a huge potential there to study bias.

A comment, a question from Jason. Hi, Dr. Jason, the other Jason here. Smile. I'm an app web product designer, but I think the XR is the future. What, where would you recommend I start to get into the industry?

So I'd say, you know, I had this, so at the end of my book, I have like 2 pages of, you know, suggestions of what to do. And one Number one thing is to go to a meetup, right? And talk with other people in that industry, see what they're up to, kind of get a feel for things, try demos. I don't know if there's— there was a lot of VR meetups in the Bay Area back at that time. I don't know if there's still that case or if your group has those sort of in-person, you know, focus on, you know, maybe some events are virtual reality, but there might be others, right? So in other words, you know, talk to a lot of people. Maybe there's— I don't know because I'm not out there anymore if that exists anymore. But if there is, get involved with the community. Try a lot of different applications. If you're, you know, a gamer, go in there and experience, you know, different things. And even if you're not a gamer, right, kind of see what's possible. Start thinking about how that might apply to your industry or for different use cases, even if it's just a game. We've gotten some great ideas from games of, wow, what if we like use this concept for training, right? We're constantly talking about those things. And then, you know, you can do Unity for personal use for free, right? And other development tools. So just get in there and start playing, right? Go get a book, right? A tutorial or something. There's more than enough books out there. My book's more on sort of the conceptual design points versus the actual, you know, how to use Unity to implement. And so, you know, of course I'm gonna suggest my book, but I'm gonna suggest other people's books that, you know, are more specific about You know, the current version of Unity, for example. Right. And also lots of resources out there.

And also with Unity, there's a lot of, uh, um, packs you can get that have, uh, predefined objects that you can add to your environment so that way you don't have to build, um, everything from the ground up. Oh, absolutely.

That's what I love about— there's also, I've seen a few that like, as part of the, you know, their asset, they provide like a tutorial that You know, okay, how to move assets around, you know, how to move components, right? It's amazing.

Yeah. And you can get assets that are rigged so they like people walking and, uh, or vehicles that bend at joints, you know, if it's like a construction vehicle, things like that.

It's amazing. Yeah.

Yeah. A question in the audience from David: what impact does touch have in VR training?

Yeah, so when we first started our, uh, you know, focusing on first responders, That was a grant from the National Institute of Technology, Public Safety Communications Research, that, you know, were the early days that allowed us to explore use cases for virtual reality, but specifically use cases of adding physicality. So we were exploring a lot of bringing in equipment like real-world equipment or 3D-printed equipment with, you know, sensors and such attached, different types of sensors, and a lot of experimentation And so that was focused on the touch, sense of touch. And that was really fascinating research. And we were getting feedback from first responders that, you know, it's a physical, you know, a physical job, you got to have touch. And so, you know, some ways that's really important. There's also a lot of trade-offs, right? Because as we're building this equipment, things can go wrong. And so we had to weigh a lot of different factors. And part of that is deploying deployment, right? And so we simplified our hazmat training. We printed 3D, um, we brought in real-world air monitors, attached trackers to them. We 3D printed our own monitors, you know, added our own buttons to it to, you know, match other things, or, you know, all sorts of, uh, things we did with. But we ultimately ended up simplifying it as much as possible because we determined the, um, the cost to deploy that in the field would be astronomical. And so I think Certainly from a research perspective, it just wasn't feasible from a cost perspective. But in the future, I hope that it is because that can dramatically, right? So if even if you only touch a couple things, I touch this desk and there's a real physical desk where I'm seeing the desk in virtual reality, it's mind-blowing. You don't have to touch everything. You just touch a couple things and then your mind just auto— well, if this is real, that table over there is probably I don't have to go touch it, right? I just assume that it's physically there because this is physically there. So you don't have to model everything. If you've ever done like the void type of experiences or other location-based stuff, it's phenomenal. Even though, you know, it's not perfect, none of this stuff is perfect. It, when done well, your level of presence is just taken to a whole other level. So yeah, touch can be very important depending what you're doing. If you're doing surgical simulations, for example, touch can be extremely important. And so haptics is commonly used there depending on what you're trying to do.

A question from Carl, our SIGGRAPH person. How do you trade off depth, breadth, and fidelity? Is breadth most important to deal with a variety of rare disastrous cases?

Yeah, in my opinion, this goes back to another thing my academic advisor preached was do it really well for You know, single user. By single user, he meant you know specific target audience, not just one person. But you know for a very specific thing before you try to make others happy. And so I'm kind of opinion it's better to you know if you try to make everyone happy, you sort of make nobody happy. Now if you're at a bigger company, right? If you're at Apple or you're at you know Meta, then you know you have those budgets of going wide. But as a small company, you really have to focus. And so we went pretty niche on not just hazardous material, but air monitoring. And because air monitoring, there are not any good ways to train with it. There are no good ways to evaluate for it. In fact, that makes it a challenge. So for air monitoring, there's standards, for example, selection of personal protective equipment, right? Of, okay, you have this, you know, I read you what the scenario is, then we have, you know, different types of protective gear, different suits or whatever up on the table. Okay, which, which gear would you choose? You choose the gear and either you're correct or you're not. It's really easy to evaluate if you know, know the answer or not, right? It's, you know, it's a multiple choice question. And so you can evaluate that. And there are standards for that. For air monitoring, there's no standards because there's no way to evaluate it. It's basically an instructor. Did you sit through the class? And, you know, maybe they give a test But really it's the instructor saying, yeah, the person knows this, right? Because it's about physical behavior that you can observe someone in the field in a dangerous situation, and an instructor has a pretty good idea. Yeah, he knows what he's doing, but there's not any objective measures. And so that's why we went super niche, because that's something that doesn't have a solution. Now we probably will, you know, we can expand out to the personal protective equipment example. We can add that as additional capability. It's just not a pain point, but we could add it because as long as, you know, you already bought the VR system and such, then, you know, we can add that for, you know, super low cost and just make it part of the experience. But the point is adding those sort of lower priority things at a later point after, you know, solving for the core problems.

A question, Dr. Jerald. I think you said your advisor was Fred Brooks. Is that the Brooks who wrote The Mythical Man-Month?

Yeah. Yes. Amazing book. So when I reread that book after many years, I realized, what was that, 1970-something, '73, '76 or something, but definitely the '70s. Like that book, he doesn't call it agile, you know, methods, but a lot of what he talks about were like agile approaches. It was just crazy thinking about that from that perspective when, you know, everyone was doing sort of waterfall approaches. But yeah, he, he was just amazing. He passed away recently, but he contributed in so many ways, you know, not just virtual reality, but many other ways. And just, yeah, his insight has been amazing.

Sounds very exciting. Other questions from the audience? So have you ever had anything inspired from, uh, science fiction or movies? You know, Ready Player One had a— or I think some places had, uh, interesting treadmills that people would walk in that aren't just in one direction, but, you know, forward, backward, right, left. Or I just wonder, you know, maybe it's not practical, but things that could be invented or could be used if there was a need for it.

Yeah, absolutely. I mean, it was— I've been inspired by those movies. Like, so, uh, you know, I'm older, so, you know, the holodeck, right, with Star Trek.

Oh yeah.

I think was an inspiration to us all. And actually, if you look back at the history, um, in the '80s— someone is saying something about stereoscopic vision, right? Like stereoscopic, you know, static obviously displays were around back in like the 1860s, I think, when they started doing them. Then the 1890s, they went viral, right? They didn't have social media, but they had, you know, regular, you know, media. And, you know, right, people were going crazy about it because it was the coolest thing ever to look inside the scope and see something in 3D, right? And there's others from like, you know, the 1930s and '40s, right? So, uh, flight simulators actually, um, you know, you could argue is, you know, I guess depending how you define it, can be a form of virtual reality. So, uh, the Link Simulator back in the very early '30s, I think actually 1929, I have a picture from— I didn't get a chance to meet him because he passed away many years ago, but his sister had sent me a picture. I randomly ran into a conference. You never know who you're going to run into at a conference. She— Edwin Link was his name, but he's credited from 1931, I think, creating the first Link simulator. But it was basically a plywood, you know, make-believe airplane. You know, then they'd bring up pictures and such in it, but it was physical, you know, virtual reality. But she gave me a picture from, I think it was 1929, that it was signed by him, said 1929. It's like, wow, that's like— all the documentation says 1931. So that was, that was really cool.

Excellent.

Oh, okay. Is that better?

Oh yeah, you're fine.

Camera offsets is one thing I guess we could do for, you know, improve with virtual reality.

Right, right. Okay, I'm not seeing other questions. Did you have any other closing remarks you wanted to make? I think we're about done.

Yeah, I think— I'm not sure, you know, because I had a few concluding slides, but I think the point is that we all as individuals, right, individuals have changed. Change the world and put, you know, they say a dent in the universe. I'd like to think kind of a bump to make it, you know, positive. But we all have the opportunity to make the world a better place. One of my favorite quotes is Alan Kay, who said the best way to, you know, we're all trying to figure out what's coming in the future. But he says the best way to predict the future is to invent it. And there's been individuals and organizations that have changed the world. So why can't we change the world, right? We all have opportunity with these amazing technologies, not just for people who have the money, but to all of us.

Exactly.

So, uh, could you give the, uh, the code, the discount code for the book once more? Yeah, absolutely. There you go, SFBA25.

Okay, if anybody else wants to take a screenshot, please go ahead and do that now.

I should have put this on the same page as my contact. So the next slide has my email, which is jason@nextgeninteractions.com.

Okay.

And as I mentioned, you know, someone was asking, do we work with, you know, do we charge upfront for, for, you know, the, the figuring it out stage, right? The early stage. Yes. So typically if it's industry, but if it's more on the You know, research side, as far as, you know, co-proposing on a grant, for example, we, we never do work in isolation, right? We always work with others. And so, you know, reach out to me if you have a project you'd like to discuss or whatever.

Okay, sounds very good. Well, thank you so much for your time and your talk. It's been very interesting, entertaining, and educational. So I appreciate that.

Okay, thank you everyone.

Yeah, so, okay, thank you very much.

Thank you. We'll end the meeting now.

Okay. Yes, thank you. Bye.