VR risk training | Phillip Mangos x Jason Jerald

This is Impactful Realities. This podcast is for those who thrive in chaos and engineer breakthroughs. We're talking high-pressure industries, high-stakes decisions, building cutting-edge solutions for the world's most dangerous, demanding, and high-stakes environments to create new realities that defy limits and reshape industries. Here is your host, Dr. Jason Jerald.

Welcome to Impactful Realities. Today, I am thrilled to have Dr. Philip Mangos on the call, who is the CEO at Adaptive Immersion. And what Philip and his team at Adaptive Immersion does is they focus on the most dangerous jobs and missions through breakthrough VR training and immersive assessment technology. Perhaps most importantly is focusing on critical incidents. Practicing things in a very safe way for these dangerous situations that are really difficult or even impossible to do with traditional training. And these really tough situations that sometimes team members can find themselves in that can literally be the difference between life and death. And so not a light topic here and truly impactful, which what this podcast is about. But then also connecting that training of finding out where it's a good fit, but then also how does it relate to your business outcomes, such as, you know, are we getting a return on investing in these projects? And so, yeah, the overall impact of XR training specifically to training and how that training relates to business and focusing on real evidence and legitimate claims versus just the hype of only talking about these things. And so we're going to go through some real numbers here of whether that's projecting what you're going to get out of this as far as outcomes, but also measuring after the fact, after you've drained this data and throughout the project of continuously updating those numbers and such. So I'd like to start at a very high level of getting Philip, your opinion on where the market is headed. as far as these immersive technologies go, and where do these XR type of technologies most add value to what the market really truly needs? Absolutely.

Thank you so much, Jason. Absolute pleasure to be here. Thank you so much for having me. I think there really is a huge demand signal for a lot of different applications of XR technology. Of course, our bread and butter is in the industrial safety and Like you said, I couldn't have said it better myself, but basically with the dangerous jobs, if you're able to virtualize those jobs and those difficult or dangerous elements to them, you can really provide a means for the people to do some safe practice and repeated practice so they can get better at it before they face those dangers in the real world. So that's one thing that's on the top of my mind. I think a lot of other organizations outside of that space, especially in retail and in manufacturing, are really looking to use XR for more digital twin type of applications and to help them to visualize outcomes of the manufacturing process and things like that. So definitely a huge demand for it. And what we're seeing is a lot of growth in those areas.

I see. I'm curious, when you say retail and manufacturing, is that different than sort of safety training? It's other forms of training that isn't necessarily dangerous? Are there situations where retail and manufacturing is actually dangerous.

To me, yeah, we focus on the dangerous aspect of it. But aside from that, there's also just like visualizing like what products look like in different contexts for the retail application. A lot of architectural and design companies are doing that. And in manufacturing, when they're testing out or developing different designs of vehicles or really complex costly equipment. Sometimes it's better to do that in VR and simulate what the product would actually look like and how it would function before you go through all the costs and all the trouble to make the real thing.

Yeah, I do remember on the retail side, right? You think of that not being dangerous, but I know there was the horrible situation years ago in Walmart, I believe in Texas or New Mexico. I don't remember exactly which city, but they actually had a horrible situation of a shooter situation and they were actually trained in VR. And then the CEO came out and said, without this proper training, unfortunately it would've been a heck of a lot worse. It sounds like you, yeah, you may know more about that than I do.

We got asked about that exact same thing, the active shooter situation many, many times recently at a summit. And it, it's one of those black swan events, you know, it's, it's one of those things that happens so rare. I mean, you never can, can fully prepare for it because it, it's such a surprise that, you know, people often have a, you know, a, a freezing response, you know, fight, fight or flight type of, of response and people freeze up and they don't know what to do. So at least getting them that familiarization and some pre-exposure to those stressful situations can really go a long way, especially in that active shooter situation.

That's one of the challenges we've had is that people sort of want to solve the problem after it occurred, right? And so you really got, for these dangerous situations, you really have to be proactive so that when those sort of rare things do happen, you better be prepared and you don't necessarily need to practice for the everyday things because if you're doing them every day, you're already good at them. But what are sort of the rare things when bad things happen and how do you prepare for that? And so to me, that sort of relates of, okay, how do we analyze, is this worth doing or not? Because it's kind of hard if it's rare to do. But I know you have some tools and such and a lot of understanding of how those relate so that you can justify doing these things and why clients come to you. And so, to being able to measure and communicate both that impact of why are you doing this, but going beyond that, right? What is the ROI? So, if things go to the board of directors or whatever, then quickly look at the numbers. How can you clearly communicate that this is the bottom line? This is why we need to keep our people up to date and trained and such.

Absolutely.

And so, I guess that's my interpretation, but let's ask you in your words, why should we care, in your opinion, about— we and your clients and others in the world, why should we care about ROI?

Yeah, I think that the most, probably the most obvious or apparent thing is that you, I think you have to be able to communicate the dollar value of what you're doing to the critical stakeholders in your organization before you can even launch an XR program. You have to show what the payoff is going to be in dollars and cents. I think that that's absolutely essential, and I will be talking a lot about that as as we we go along our our discussion. Definitely got some ideas on that, but digging deeper and talking a little bit more about the domains that that we work with. I mean, we're we're dealing with very dangerous industrial safety and military domains where it's a very real risk that that people can get hurt, injured, or die on the job on any given day. And it's kind of hard to talk about ROI in that dry sense of just dollars and cents in those industries when you're really saying, hey, we are literally saving people's lives on the job by getting them that pre-exposure and that preparedness that they need to handle those difficult situations. Of course, All of that stuff can also be quantified as well in terms of, you know, how many accidents are you preventing or even near misses, which are more frequent things that happen on the job than the actual accidents or incidents themselves. And we've been analyzing lots and lots, literally hundreds of thousands of safety incidents across many, many different domains to look at that, kind of looking at the actual behavioral DNA of what happens when people face those unscripted or unprepared situations in the real world when things come together and create a very dangerous or bad situation. And they happen all the time. Definitely a lot in the manufacturing, in the construction world, basically anybody where where people are doing physical things on the job as their primary full-time thing. So, we definitely, I think it's very critical that we look at that and try to understand the mechanisms of what's causing those lapses in safety behavior. Yeah.

You know, I've seen some of those reports. You think like, what's the big deal, right? So, for example, filing complaints sometimes. For example, I remember a specific report with a SWAT team. That a complaint, and that's, those are dangerous situations, right? But even if something bad doesn't happen, if there's a complaint, it ended up costing like something crazy, like $30,000 per complaint because they had to investigate it. They had to, you know, bring in the legal team, all this stuff. And those sort of hidden costs people don't realize, there's like, you know, things you think that, oh, it's just a complaint. They actually really do have monetary value. Or in the other end. Absolutely.

And I think the good thing about—

Oh, go ahead.

I was going to say, you know, just to add to that, I mean, I think that that's also a very rich source of data because, you know, we talk about, you know, black swan or rare events that happen. It's really hard to calculate any ROI on such a small sample, right? But people are complaining and, you know, putting reports and, you know, near misses every day. Of course, even near misses are often unreported or underreported in safety organizations because, you know, no one wants to get in trouble and chewed out. But it is very rich data, and the more of that data that we can capture, we can have a better understanding of the landscape of that organization.

Yeah, and I know it also depends on culture. I know some— we were at a chemical plant, for example, at one point, And right up there near the entrance, they had the big billboard of how many days it's gone without an incident, and they really prided themselves on that. I know not every company prides themselves at that, but yeah, thinking about, okay, how did those numbers of safe days relate to a monetary value? That would certainly be interesting to look at.

Definitely.

I know on the hospital side as well, it sounds like you were saying, sounds pretty horrible, but They actually put financial numbers to deaths, right? Of this is what, you know, it costs the hospital when someone dies sort of thing. So it's pretty grim, but it's sort of the reality of having to make, you know, sometimes really tough decisions or to track, you know, your progress. How are you improving?

Absolutely.

I'm curious. Yeah, I know you work a lot on the military side, so we can, you know, talk about that, but I'm also interested in for our audience too, because they're not necessarily all military people, but where do you see the most important industries where safety truly matters, right? Of whether it's construction or manufacturing, like you mentioned, obviously military, but yeah, what are your thoughts where this is especially important?

Sure. I think everywhere. I mean, like you said, even the most unexpected places, Customer service in a retail store can result in a dangerous situation. But if I had to rank them, I would definitely say manufacturing and construction near the top. We're definitely seeing rates of people getting injured at around the 30 to 35% level within the first 6 months on the job in many manufacturing industries.

That's insane. I would've guessed like 2% or something small.

Yeah. And that's, um, it is higher. I mean, that includes everything. That's not like just the severe injuries or fatalities. That's people getting hurt enough to like, you know, take some time off work or, you know, a few hours on the job. Um, but it happens all the time and that's the most vulnerable period.

What about those near misses? Are those part of that? That's probably higher than 35% of Like coming close, right?

Yeah, that's just guesswork really, because, you know, a lot of that really goes unreported. Some people, like you said, they have a strong safety culture. They might report them and it goes to OSHA and everything just to make sure they have the right paperwork in place. Not everybody has.

Or I can imagine not wanting to report it, right? Because, oh God, I just about, you know, cut off my arm. Thank God I didn't. You know, you, A little bit of anxiety there, but anxiety's probably also that, oh God, I can't report I just messed up to my superior.

Exactly. And those moments happen so fast, you know, you have to kind of redirect your attention back, back to the task to, to avoid the next maybe not so near miss.

Yeah. And then there's other things like that are sort of indirectly but very directly related. So I, I've heard on the firefighter side, 2 of the most, uh, leading causes of death are heart attacks afterwards, right? Not actually during the incidents, but, you know, it's tough to sleep at night. And then suicide as well. If, you know, you couldn't save someone's life or whatever it might be, it's pretty horrible. And that isn't necessarily directly, you know, attributed to training and feeling like you're properly trained. You could have done the best job you've done and those sorts of things. So yeah, not everything can be converted to ROI, but certainly important. So I see ROI as one measure, not the only measure, but at least it's sort of this objective way of looking at things versus other things are harder to put numbers to.

Agreed. Definitely.

So yeah, as far as the, those critical incidents and how do we analyze those, especially when it's so rare, maybe you don't have a lot of data. for, you know, your specific use case. But I know we were talking earlier where you talk about different use cases that actually are relevant, even though it's a little bit different, they're very similar in some ways. And so you can sometimes use that data. I think you called behavioral DNA. Could you talk about that a bit?

Yeah, absolutely. And that's really a way that we've used to help basically create more robust training that can be used in a more customized way in specific jobs. It really doesn't make sense to provide training that's not, or non-job specific. You know, you really want to get down into the weeds of the technical tasks that they're engaged in on a daily basis, the more dangerous events that they might have to deal with situated in that exact type of environment and context. Problem is, it's kind of hard to get ahold of a lot of accident data or critical incidents, safety incidents, and so forth for any given job. So, what we've done is we've assembled a very large mass of critical incidents and done some AI and data mining to it to pull out the major themes and understand what things can happen in different contexts, no matter what the job title is. Married that to actual job titles from the Department of Labor, the Occupational Information Network. And using that, we were able to kind of create the behavioral DNA of specific job tasks that are often conducted within specific jobs and quickly create new scenarios for those jobs, even if there aren't a lot of examples to pull from.

Okay. So that, it sounds like, is that correct? That sort of helps Define what are the important tasks to focus on. Because in this industry or this use case, you saw it's important, so it's very likely to be in this case.

Exactly.

Maybe a different industry, but very similar use case.

Exactly. Creates, basically creates a blueprint for custom training and more exact training for a specific job situation, thereby allowing you to increase the repetitions and sets for—

Yeah.

for practicing in that specific job.

So I don't know, our audience, I suspect, may not get how important what you just said is. For us, that's been a major challenge of, okay, for this use case, how do we collect enough data? And perhaps the biggest challenge we've had of why is this worth doing this? Right? It's just, it's hard to get access to that really specific data. If you're working closely with a large organization, sometimes you can, but not always the case. You know, a lot of times it's private information you can't get access to and such. And so that is huge, that concept of, okay, very similar, you know, use case, maybe task analysis is very similar and different industry, but really kind of doing the same thing, the same types of dangers and such. So that's huge. I just want to emphasize that.

Definitely, definitely agreed.

Now, I also, what I love learning about you is, you know, it's kind of, we have a lot of, you know, anyone can talk about how great these technologies are because they are awesome and amazing and all of that, but we can't really rely on hype anymore. 10 years ago, right, VR was sort of this new thing, at least as far as mainstream. It's been being researched, you know, 40, 50 years. It's amazing looking at the history of it, but sort of that mainstream experience with VR was like, oh, this is so cool, let's pump a bunch of money into it. But what I like about sort of your background is looking at the actual research, scientific-based design concepts of how to put these things together. So you make sure you're focusing on the right things and doing it on the right in the right ways, right? Because if you, for example, if you train people in the wrong way, that's worse than doing the training at all. So you better make sure you're doing it right sort of thing. And so that very sort of rigorous approach of iterating continuously based off what you find, the user feedback, and balancing that research, you also have to be a little bit careful because you can go too far to the research side. Academics do a really good job of that, but it's not necessarily You know, directly applicable immediately. And so it seems you found a really good balance of, you know, the business, but based on that science, those scientific processes. And so could you talk a little bit about that, sort of maybe going a little bit to your background and how that applies to how you're using that scientific basis applied to business, not just for science sake? And not just, you know, making up numbers to try to get the sale, but for having real, real impact.

Definitely. Yeah, I think it's absolutely critical to have a science-based approach in everything that we do in training, especially when, you know, danger and lives are at stake and everything. So, I mean, it definitely helps to tell a story about how impactful your training is going to be and to put that into numbers. And there definitely is some scientific guidance to support that. My background, I've been doing this for many, many years, 25-plus years. And my background's in industrial and organizational psychology, which is very much about predicting performance on the job and helping improve performance on the job. Did that for many years initially with the Navy. Helping naval aviation trainees make the most of their training, analyzing their jobs and developing the training for them. Moved into the pre-hire assessment world and did a lot of work with large-scale testing, basically for organizations doing massive amounts of hiring, making sure that those hiring systems were predictive and fair. And resulted in ROI. That's really the roots of a lot of the ROI analyses that I'll actually be talking about today. Way back in like the 1940s and '50s, the military started using those new hiring technologies to— or testing methodologies to get the right people in the military, and they wanted to quantify ROI. in terms of dollars and cents of these new selection systems. That same thing works almost exactly in the same way with training. You're replacing the training impact, or you're replacing the selection impact with the training impact, and the numbers and all that work the same, and they're really helpful for telling a story.

Yeah, I love that. It seems like, you know, XR is sort of, even though it's been researched for many decades, as far as actual application beyond the lab is sort of everyone thinking we need to reinvent everything, but we don't. Like a lot of these processes and these different ways to analyze things and such have been there. It just happens to be that we are applying them to, XR happens to be the tool, but the actual measurements and such, there's Actually rich history there. And so, definitely, I'm curious why for you, before we get into, you know, getting into specifically how you do that, I'm curious what was it for you that decided to start focusing on XR? Because you weren't, you know, you were looking at more of the traditional approaches to, you know, this measurement and analysis, but why did you jump to XR?

Absolutely. Number one, I think it's just a very cool and a very effective technology. I have an affinity for dealing with dangerous jobs since I've been doing it so long, both studying them in my professional capacity and doing them before I became a professional. Basically since the age of 14, I've worked in different environments, including a manufacturing facility, Where at 14 years old, I suffered my first injury. Wasn't too bad, very minor compared to the other injuries that we look at every day now. But several family members and friends and everything did work in the same kind of environments and had similar types of problems. And then working with the Navy, you're dealing with fighter pilots and special forces and stuff like that. It's just there is absolutely a cool factor. And there's a helping factor of knowing that you can actually save someone's life by giving them good training. And firmly believe that XR is one very good way, not the only way, but one very effective way to do that.

Yeah, it's interesting. I think as we mentioned this when we were talking earlier, how it was interesting, you and I seem to have very different paths, but kind of ended up in similar situations. I started off because I thought it was cool and interesting and amazing. And then I started, you know, figuring out, okay, how do we, you know, actually analyze what are the best fits, all of that. Whereas you came kind of the opposite direction of doing the analysis, you know, where is most important to do training and then adding the cool, you know, discovering the full cool factor afterwards. And so, yeah, interesting that you can have very different journeys, but sometimes the destination can be, you know, 2 very different people end up in the same spot.

Oh yeah.

I would love to get into the details of, not too low-level details, but enough for our audience to kind of understand how can we project what the effectiveness and the ROI and such will get out of this. And I know you have some slides that are showing real numbers versus just, you know, us talking, but, you know, mathematical models that show what the ROI can be. Before we dig into those slides, I think it also kind of depends who your target audience is, right? Because if it's the guy on the floor, he may not care about ROI, right? So there may be different reasons why he's doing it. Like maybe it's, you know, he feels more properly trained, he's not getting hurt, you know, those important things, but he's probably not thinking ROI, but others are very much thinking ROI. So before we get into, you know, kind of explaining your model here you've created, What are your thoughts of who this model is, even if it's not for our audience, but if our audience is speaking to, you know, to need to be able to justify a project they have, they need to, you know, speak to the stakeholders or whoever to justify moving forward on the project. And so the question here is, who would this sort of target audience be for this ROI analysis that you do?

Sure. I think it would be very good for pretty much anyone who's dealing with or hoping to implement a large-scale XR program into their organization. So definitely L&D professionals and even HR people who, you know, we're doing XR for personnel assessment and selection applications. Beyond that, I mean, the people that are actually making the financial decisions to invest in these technologies, they need to be able to understand how much of an impact they're going to have and have some hard data with it. And finally, I would say supervisors and management of those dangerous jobs, because as I'll be talking about, you have to have some really core pieces in place to make the training work and to be able to effectively measure its impact over the long run. You can't just make some numbers and throw them into this Excel that I'm going to share with you. They have to come from some real performance data. So, that would also include maybe, you know, more HSE and, you know, safety program professionals in these organizations as well.

Okay, excellent. I'd also say, for our listeners, just because you're not one of those people doesn't mean this is not useful to understand, or at least understand the basics. because you need to communicate, right? Think of you not being the target audience, but who is your target audience? And so if you're, you know, maybe you're a developer, right? Instead of just blindly being told what to do, if you're excited about a project and you think it has true value, you need to at least say, hey, we should look at the ROI of this. Maybe we should look at this model so that you can kind of justify what you're, those projects that you think are important. And then you're at least able to communicate with that. target audience, those stakeholders.

Okay.

With that, let's go ahead and dive into the slides you have, Philip.

Sure. Yep. And I'll just preface it a little bit, you know, just an example that I've kind of been thinking about to make it real for our audience. And like you said before, there's many different forms of ROI. And one example that we have is we develop actually some very specialized training in the military for people who have to go and investigate a downed aircraft. And this is usually a live training situation where you gotta send a whole bunch of people, a large group of people, pretty much from all over the United States to a single training location. And, you know, all their specialized gear and, you know, there's a lot of logistics and planning involved. It's only a single site, so there's like very limited opportunity for each person to get enough training, uh, training reps and sets. And that's kind of the opposite of what you need to do to get good skill acquisition. You need lots of reps and sets. But these people, they might only get like a single run through the exercise. So, we're putting them in a virtual environment. We're hoping to increase the number of opportunities for them to repeat the exercise in the virtual environment, get more practice, make it more safe and everything. So right off the bat, I mean, you're saving money off the material costs, logistics, the travel, the planning and all of that, the maintenance for the live training. So only looking at those factors, it's actually pretty easy to quantify the—

We've been told on the firefighting side, there hasn't been materials, the actual live training with the actual release of the chemicals, right? But there's only a few places in the United States where you could do that training with those live agents. And you might only have that opportunity if ever, right? Depending if your department has funding, you might only have that opportunity once every 5 to 10 years. And so, you know, maybe a couple times in your career and with the immersive experiences, you can practice that as much as you want. And so clearly there's opportunity there for, you know, cost savings and not just cost savings, it just may not be possible at all. So getting the training or not getting the training. Okay. With that, let's go ahead and jump into your slides. I'd love to see the specifics.

So what I wanted to share with everyone today, I mean, you know, we talked about these basically very quick and easy ways to do ROI just by virtualizing something in an XR experience. But what I want to really emphasize is something that's not thought about a lot, and that is how much improvement in your employees' job performance you're realizing with the training. And a lot of those factors that we talked about, increased repetitions, More safe practice opportunities and everything. Those can all result in a more permanent level of learning on the job, you know, allowing each person to be more effective, more productive, more adaptable, more responsive to safety emergencies and things like that. So what I want to talk about specifically is the ROI of job performance. And there's 3 basic inputs needed to calculate this. The calculation is really easy. The way that you pres— can present the results to stakeholders is also very easy. The hard part is having a framework in your organization where you're doing, where you're actually correctly measuring the impact of the training and the training alone. And not confusing it with other extraneous effects of that, that could, you know, just spontaneously cause someone to perform better or worse on the job. And also have the right performance metrics in place that are valid and reliable and that can be used to, to actually show the monetary value of a person's job performance. If you don't have that system in place now, what you can use as a proxy for that is basically a person's salary level, the amount of salary that they're making on an annual basis. So when you think about that, you're paying somebody just to use $100,000, $100K per year as just an example figure. And that person is an average performer. They're performing right at the 50th percentile. What I wanted to focus on really is the impact on someone's job performance. And that's really, to me, the whole point of training. You're really investing in someone's future on the job. You're giving them enough learning material and content to be able to hopefully permanently move to a higher level of skill acquisition. perform better, be more productive, engage in better, more adaptable safety behaviors on the job. And there is a quantitative value to that that you can actually measure with if you have the right metrics in place and the right framework in place. The actual inputs of that are very easy. It's just, a very simple formula that can be done in an Excel sheet like I'm, what I'm about to share here. The hard part I think is actually developing that systematic framework of how do you measure someone's training performance? Are you using the right assessments to capture their performance just like you would on a test at the right time? And do you have the correct job performance measures in place in your organization to systematically and frequently measure someone's performance on the job. Just going back to that manufacturing example, that could be something as simple as their productivity, how many widgets that they're producing per shift. If you're framing it in terms of safety, how many accidents or near misses they have avoided. And for any job, it could be well-developed supervisory performance ratings where you have supervisors or peers and the self-performing ratings of their performance along, along different dimensions. If you can do a good job of all of that, then the ROI calculation is, is very easy, and it works both for XR training and any other type of training, live training or whatnot, that you'd like to implement to improve performance.

Yeah, I want to say what I like about these models, or maybe, you know, you could say don't like about the models, but sometimes you have to make tough decisions to proceed or not. And sometimes the answer is it's not worth proceeding, right? So if, you know, all the costs and effort that go into something aren't sufficient here, you either need to figure out how to add more value or say, you know what, we have to make the tough decision and not take on this project. And so it's really a decision-making tool, not just, you know, a way to validate what you're going to already do anyway.

Absolutely. Yeah. And there, there's actually an input that gets at that exact issue. How much of an improvement is your new training system going to provide over the existing baseline or the absence of training? So to kind of jump into that a little bit deeper. Oh, and one thing I also wanted to mention, if you don't have all those performance measures in place, you can also use someone's annual salary to conduct the same ROI analyses. And actually, that's exactly what I've done in this example. And if you think of someone with, just using an example number of $100,000 of an annual salary, if they're an exact middle-of-the-road performer, they're performing right at the mean among all their peers in the same job, you're basically paying them $100,000 and their expected output to the value would be roughly that same amount. But somebody one delta above the mean, is going to be producing roughly 40% more output. So you're paying them $100,000 and they're producing just that one person alone by performing one standard deviation above the mean, better than about 65% of their peers. They're producing an additional $40,000 per year alone, just that person.

Wow.

Yeah.

There's also all that overhead, you know, per employee that you have to take into account. You're going to be losing money if, yeah, they're only their direct salary they're making up for.

Exactly. So the 3, uh, just at kind of a high level, at a high level, the 3 inputs for, for conducting an analysis of the ROI of job performance. Is how much additional validity you're getting or how much better prediction of someone's job performance with a new system. So if you're not doing any training at all, it's almost like flipping a coin whether the person's going to get hurt or not, or whether they're going to be a top performer. It's pretty much random chance. The better training that you're using, the more reps and sets that you're giving them in XR, for example, then their scores on that final training assessment are going to have generally a positive correlation with their job performance. The higher, the better. And that's on a 0 to 1 scale. We'd be very happy to have maybe a 0.3 or a 0.4 for that. training validity. But basically the first input is how much extra validity you're getting with your training system. The second one is basically how much of a delta there is between successful and non-successful performers. on the training. And that has to do with the cut score or the benchmark that you're saying, hey, this person has to get, you know, at least an 80% on the training because we know that that represents or that's correlated with zero safety incidents over the first year. So the bigger that spread or that delta is between the passing and the non-passing performers on the training. And that can all be measured with a final assessment or a series of assessments that are conducted throughout the training. Uh, the final input is how much of a spread there is in people's job performance. And basically what that is getting at is the complexity of the job. Um, for very easy jobs, it's very easy for everyone to perform pretty much exactly the same way. And if you can think of a very simple assembly task where there's only one correct way to do it and it takes a few minutes to learn it, if you look at 20 people, 20 men and women doing that task, there's not going to be much of a difference in their job performance. Some people might be faster or some people might be slower. But the trend we're seeing now in manufacturing and construction is people are dealing with autonomy, people are Having to make very fast and complex decisions all the time. There's much more complexity on their job. So that variability in job performance is much higher. And that is the third factor that affects the ROI.

So as far as that deviation, do you see it more often? It's more like getting the low-level performers up to some standard. Versus, you know, the best performers are kind of already where they're at. Maybe you're not going to improve them much, or maybe there is cases where the best performers, because they're so committed and so passionate or whatever reason why they're good, that maybe they see a large increase. Have you noticed any trends like that?

Yep. Yeah, I mean, definitely the goal is to get everybody more uniform in terms of like core tasks. So you want, people's performance to go up and to have— be slightly less variable. But then there's also the skill decay issue. You know, some people who have been on the job for many years and consider themselves experts, no one is completely invincible to skill decay. So there's always a need for training.

Yeah. One thing I noticed as well is it's not necessarily experience. Makes you a higher performer. Sometimes, you know, you can be complacent if you've been at some job for 40 years, you think you know everything, and then you actually watch and the younger guys come in and they're, you know, because maybe they're, you know, an open mind to doing things a different way or whatever. And so, yeah, it's not necessarily, at least what I've seen, the most experienced or the most skilled. It's probably correlated, but not necessarily in all cases.

Definitely. So with those 3 inputs, you can basically produce an estimate of how much ROI, annual ROI, you'll be able to realize per employee. And then you can just simply multiply that by the number of employees in the job that you have employed or are hiring, you know, in a certain timeframe. So in this very, very simple example here, you know, we have the training validity at a level of 0.35, and that's above like the almost, you know, we're assuming that the organization or the job did not have any training in advance. So this is going to be a number very close to 0. We'll say, you know, 0.05. So that improvement is about a 0.30. That's a very, very good improvement. We have a delta. Yeah, that's a big difference. Yes, yes indeed. Got a delta between the high and low performance on the training of almost 1 delta. It's 0.7. The measured variability on their job performance. This is something that can be done very easily if you just get a large sample of people on the job and whatever measure of job performance you are looking at, you take the variability or the standard deviation of that. We're using $20,000 based on a $100,000 salary, and we're assuming 1,000 people. ROI per employee is roughly $4,000 per year, resulting on the high end, $4.1 million for the organization. On the low end, $3.5 million. And that's affected by where you place the cut score for saying whether that person is proficient or not in the job. That's why you got to have those performance measures in place. You got to be able to make decisions about whether the person is qualified.

Yeah.

To continue performing on the job or not. And if you can do that, you can get an actual range of how much ROI you're buying with this training system.

So I'm curious by ROI, so right, the revenue on top divided by the cost. So, or maybe it's, you think about it differently here. Is there an investment number that goes in here? As far as strictly the ROI percentage, I guess it would be this, your number on top of what are the extra costs. Does that sound correct?

Exactly.

You want it in percentage form?

Yes. This is just one piece of the puzzle. This is just the ROI of job performance. It's not taking into account the cost of implementing this new training. If it's going to cost half a million or a million to create this new XR training system, it's basically saying that you'd get a 4x ROI in the first year.

Okay.

Makes sense. Just random numbers out there. And of course, it's not taking into account the cost savings and the overhead and the saved travel costs and logistic costs and everything like that. And of course, every new XR training system also has its own maintenance costs as well as you're developing new software and hardware and the like. So those have to be factored into the equation as well. Really what the last thing I just wanted to talk about here is how do you communicate these results? Let me go into the slide mode here. You say, okay, we're going to create all these new, uh, we're going to create a new training system and realize over $1 million or $3 million in cost savings and improved job performance. So how do you communicate all that to your stakeholders? Well, if you look at the raw data and you say, okay, we have a positive correlation, you know, as your training performance goes up, so does your job performance. Everybody wants that. But you tend to come up with this very difficult chart. And I would ask our audience, you know, what's easier to look at and talk about to your stakeholders, these thousands of dots and slanted line, or a simple chart that says, hey, everyone who passed the training or got a green on the training, their average dollar output is going to be about $110,000 compared to the people who failed the training per person, $99,000, realizing an almost $10,000, yeah, about $10,500 ROI per person. We've used this template or very similar variations of it. Sometimes we have like a red, yellow, green to show where they lie on their training performance and then assign each group of people who achieved that level of proficiency a dollar value to show how much ROI you're getting per person for their job performance.

Yeah. What I like about this as well, I guess similar to what, you know, we briefly discussed previously, is it sort of tells you, well, we need to get those people that are failing. It tells us what we need to focus on. And so we need to reduce that deviation or we need to get them up to a standard. And if we increase more people up to that standard, that directly maps to the money.

Definitely.

Yeah.

I mean, this provides a direct roadmap for doing that and also making more tweaks into the training itself? Like, where do you set that cut score for defining who's proficient or who's not proficient based on their training?

Yeah, I can imagine. I mean, that's one of the powers of XR is you have all this data, right? If you just go out in the real world, maybe you're watching someone, maybe you video record them, but you're not getting specifics of what actions they're taking, objectively measuring what they're doing. But with XR, because you're tracking pushing buttons or grabbing things or just body motion and such, that you can see where the problems are. And so it makes it very clear, everyone seems to be failing in this area. Okay, well, that's what we need to iterate upon.

Exactly. And that's actually a great point that that's something we dealt with a lot. It's actually pretty pretty effective to measure someone's job, someone's performance in the simulation environment. Because like you said, there's so many data streams to pull from and you can kind of get at every aspect of their decision-making, their reaction time, all the actions and decisions that they do. The hard part's measuring their performance in the wild, like—

Yes.

In the actual job itself, because there's so many outside contingencies and If you're doing ratings, you know, the supervisor doesn't always remember, for example, they don't have the cognitive capacity to remember every detail of what the person did over the last 3 months. It's good to have objective measures where possible. That's looking at, you know, people's output and productivity.

Yeah.

Even those can be flawed, but basically, you know, the more and the better data that we can get in the wild, that would really help all this ROI analysis.

Absolutely. And I think that's a reason to work with, you know, others that aren't related directly to the implementation of the project of having that external validity. Okay, they're doing great here, but, you know, you could argue, well, that's a video game, you know, of course they're doing well. Is that the real thing? So you need to actually collect that real-world data to see if it holds up and they're doing the same thing. You know, if it's no longer a game or nobody's watching them or all these different confounding factors that come into place. But if you can work with those individuals and really understand outside of the XR, it's not just about the technology here. That's essential to work with that diverse team from those different perspectives and different forms of data and such.

Definitely.

Which I guess comes into play with you being a, you know, having that that scientific background outside of XR, I imagine you understand all those non-XR things of measuring in the real world as well that most XR technologists, they don't know how to do that.

Yeah, that's absolutely the case. And I myself and most members of my team are human performance professionals before XR professionals. And In order to do all this, I think you need to do a really good and deep job of all the basics, the task analysis, the cognitive walkthroughs, having a deep understanding of the task before you create the training, measure everything, make sure everything is as reliable and predictive as possible. And there's absolutely some well-structured and science-backed ways to do all those steps. It's the stuff that many people see as like the foundational stuff, but I honestly think it can't be skipped if you're, you know, you're creating training for a job where people can get hurt really bad. You know, it better be accurate as much as possible and as realistic as you can make it.

Yeah, absolutely.

We try not to take shortcuts on that front. We really, especially in the task analysis phase, we actually go out and shadow people on their jobs and really understand from basically a first-person perspective, if I was a worker on the job, how I would have to do those same tasks that we're modeling and putting into the VR.

Yeah, I guess that really emphasizes the importance of diversity, people from different backgrounds and different skill sets. If you just put together an amazing, you know, team of VR developers that had some, you know, major hit game and they're the best there is, that doesn't necessarily gonna mean you're going to succeed with your training project. You need to really understand—

Definitely.

What are the core challenges? Ideally, getting those people on the team instead of just going out and talking to them. In our case, that's how some of our most innovative stuff was working together. They didn't understand, The subject matter experts didn't understand the technology, and then our developers, at least especially at first, didn't understand the needs and the pain points and the specific tasks and such. But by bringing those people together, you really come up with these just spectacular, completely new innovative ways of doing things.

Yes, absolutely. Quick thing here for anyone listening that'd like to contact us, follow us on LinkedIn, and any of the materials that we talked about, that I talked about, I'm happy to share with anyone in the audience today as freebies. Just feel free to reach out. Excellent.

And we will, you're not able to click on this because this is just in video form here, but in the show notes, We will definitely include this. So you'll just be able to click on it instead of having to look at this and type all of this out. And as far as reaching out, Philip, is that correct that, uh, it looks like LinkedIn is probably the correct, the best way to your preference?

Absolutely. Or, or email as well. If, uh, if you, if, I don't know if you normally supply that, but we'd be happy to provide that as well.

Yeah, we can certainly put your email in the show notes as well.

Cool. Okay.

Well, that, thank you, Philip. That was a very detailed sort of analysis, a way to look at not just what is the results, but how to iterate and improve upon the results, right? Instead of just looking at the data at the very end. Now, before we close out, I kind of wanted to ask one last question of, What is applicable today to our listeners? What you've seen over the last 6 months, has there been much change? Obviously, you know, there's technical things that have changed. I don't need to mention the 2 letters that are a very big deal right now. 10 years ago, the letters were V and R. Now, of course, they're A and I, but also, you know, There's been changes in the government, for example, whether you like it or not, is just the reality of what things are the way they are. And especially if you are working with the government, you sort of need to justify your work. And I think that's becoming a common trend. Again, whether you like it or not, it's the reality that if you want to continue your program, you need to justify why it's important to improve that program. As just some, you know, examples and That's my perspective. But what have you, Philip, in your opinion, what have you seen over the last 6 months or happening now that was maybe different than a few years ago?

Definitely. Yeah, we're definitely seeing, like you mentioned, the new generation of AI models and the applications that are coming out of that. are really providing some really cool capabilities and everything. But behind all those shiny new tools, there's a lot of boring stuff or stuff that may seem boring, but it's actually very essential that I think is driving some changes in several industries over the next 6 months and beyond. So who's going to power all this AI? We need these huge new data centers, which is creating more more jobs in construction. We're seeing huge trends in more construction projects and manufacturing. Manufacturing, however, we're having a shortage of people. It's hard to recruit people into new manufacturing jobs. Already a small applicant pool, and it's hard to get people excited about joining the manufacturing force. Workforce, that is. But we still need to build a lot of new stuff and cool stuff. So I definitely think that trend is going to be continuing over the next few years at least. And we need to get XR technologies to support people in those jobs and make them more proficient. Another thing that we're seeing is in the defense industrial base, We still have to maintain and sometimes replace a lot of the legacy equipment. We're always building new aircraft carriers and fighter jets and everything like that, but we have a lot of new modular stuff, especially in the unmanned world with small handheld drones. So I think a lot of new trends in manufacturing and autonomy to support the defense base, as well as some more boring stuff like getting the rust off of our— off the ships that are out at sea for long periods of time, but that is still very essential and posing a lot of dangers to the actual maintainers and construction personnel that support that.

Yeah, that's one thing I'm especially excited about, even with the boring aspects of it. Is, I guess some people would call boring. I don't necessarily think of it as boring, but combining everything together, right? And not in just the obvious ways of just throwing buzzwords together, but just when you're throwing these different technologies together, it's just crazy things that you can like come up of innovative solutions of really having that impact, not only because it's cool, but actually having that effect on the world and improving in significant ways, not just, you know, a couple percent improvement, is by bringing those different technologies together. And we are certainly seeing that now with just, you know, how everything fits with AI. Drones maybe aren't as much of a buzzword now, but I'm especially excited of seeing how robotics mixes with AI, right? Because, and how then humans interact with those robots so that, you know, you sort of have this—

Definitely.

hybrid approach. And so exciting to think about all of that and how the world's going to change. And our goal here is to not just look at those trends, right? And follow the trends, but create those trends and create that impact on the world. It's of how we want the world to look. And so it's up to all of us to decide and take the action to do those things.

Yeah.

Yeah.

And that's funny. I mean, I don't mean to say that any of those jobs are boring at all. It's just they often don't get as much publicity or media attention, more behind-the-scenes work that makes everything else successful. So actually, we should be honoring the people that are doing the ship maintenance and aircraft maintenance and everything like that.

Yeah.

They're really doing some really critical work.

The XR industry is 100% guilty of that, right? All this, at least in the past, all this hype and how cool it is and all of that, which is good in some ways. It helped us move forward. But I actually think it's good people are saying, oh, it's not growing as fast as, at least because everyone looks at the entertainment side. It's clear it's growing in sort of the industrial use case side, different things that aren't talked about as much. But I think it's in some ways healthy that there's not all that hype and expectations of just unsustainable growth and all of that, because that forces us to look at where is it actually of value instead of it's just cool, right? Cool is a great thing. We can't complain about that, but we need to move beyond it only being cool. And Phil, you and your company are definitely doing that.

Definitely. Definitely. Yep. Definitely hoping to do so.

Okay. Well, thank you for your time and definitely any of our listeners feel free to reach out to us. We will provide in the show notes Philip's email as well as his LinkedIn. So you will be able to connect with him there.

Impactful reality requires more than a conversation. It takes action, research, And collective wisdom. If it made you think, we'd love to hear your insights in the comments, or contact us directly to shape the future together. Don't just subscribe, pass the torch. Put it forward. Send it to the next one in your circle who manifests things that matter, the builder who dreams big and innovates towards more impactful realities.