Improving Manufacturing Processes through Augmented Reality and Virtual Reality: An IME West Preview

Good afternoon, everybody. Welcome to the Industry 4.0 Club's special edition of our fireside chat. The club's purpose is to engage the power of diverse worldwide talent to accelerate the global evolution to Industry 4.0. We accomplish this by hosting and supporting respectful, thoughtful, and interdisciplinary open conversations and creating quality content on wide-ranging Industry 4.0 topics to expand the reach of our community. We're glad you've decided to join us for today's fireside chat with Kelly Malone. Kelly's going to share his perspective on his upcoming panel on virtual and augmented reality that will be taking place at IME West. My name is Mike Ungar, and I'm an Industry 4.0 Club co-founder. For the past 3 years, I've been a Focal Point business coach and executive coach. Prior to that, I spent 35 years at Michelin North America in a variety of continuous improvement, operations, finance, and human resource roles. So let me introduce our guest today, Kelly Malone. Kelly is responsible for technical pre-sales, project delivery and services, and customer support and account management for Tactile. For more than 25 years, Kelly has been in— working in technology, primarily focused on software operating systems development platforms and software application commercialization. He joined Tactile in 2015 following his role as Director of Business Development at Microsoft supporting the Windows platform team and specifically HoloLens. Kelly spent more than 10 years in a variety of sales, business development, and product management roles at Microsoft, and also held the sales vice president role at Accenture and Spring Wireless. So Kelly's going to be participating with us in the advanced manufacturing educational track at the upcoming IME West show, February 7th through the 9th in Anaheim. As I mentioned earlier, he's leading the panel titled Improving Manufacturing Processes Through Augmented Reality. And virtual realities. Kelly, with that introduction, welcome. We're glad you could join us today to talk a bit about virtual reality and augmented reality.

Thanks, Mike. Yeah, I'm looking forward to it.

So it's a— just to kind of get us started, I think a lot of folks still get confused between what augmented reality is and virtual reality. Can you help us sort that out a little bit just to get us started?

Sure. Well, now throw in the term the metaverse and then everybody's thoroughly confused, right?

Yes, absolutely.

Yeah, I mean, typically, you know, virtual reality refers to experiences and devices that fully immerse an individual, so their environment is completely transformed, you know, typically through some digital means, you know, like computer vision recreation of another world or another environment. It can also be— a lot of folks are familiar with like 360-degree video experiences where a camera has taken, you know, a 360-degree, perhaps even, you know, 3D type of environment, and then somebody's transported to that in a virtual you know, VR headset.

Yes.

So that's kind of the VR experience. And then augmented reality really refers to the augmentation of your immediate actual environment, right? Whether that's overlaying digital information on top of your field of view of that environment, which can be through a camera, often, you know, even on a mobile device, you can hold the camera up or the device up and through the camera and the screen, it'll overlay digital elements on top of that real-world environment that you're in, or even like spatially anchored content where it's actually placing things on the tables or on the walls or within your physical environment and somehow augmenting or transforming that physical area that you're in. And there are definitely different levels of capability depending on the devices and the use cases that you might have.

You know, that's As you describe both of those, I'm wondering, how does virtual reality have a place, let's say, in the manufacturing environment? It seemed to me if you transported away from it that you wouldn't use it there. Am I off target in thinking that way?

I mean, I would say you can actually use virtual reality in the manufacturing environment. I would say the place where it's primarily used today is in training. So think about somebody who's in a classroom or perhaps, I mean, they could be in a hotel room or an office, but becoming familiarized with a new manufacturing plant or with manufacturing equipment or even an assembly line, which it might be costly or expensive or even dangerous to have them learning in front of the physical equipment before they get some familiarization training. Or something like that, you know, or you might be able to train a lot of people at the same time when you may not be able to get that many people in front of the physical equipment. So VR definitely has some capabilities there, and increasingly the virtual reality headset manufacturers are enabling what's called pass-through video. So they're turning those VR devices into augmented reality devices by allowing the camera to let the real world into the virtual world and manipulating it that way instead of, you know, historically where you're using the camera to overlay content on top of the real world, or even see-through glasses with what they call waveguide optics that allow you to interact digital objects through your eyes, seeing the actual world just like through a pair of regular glasses or Or what have you.

Yeah, that's mind-boggling, I guess, because I've always kind of preferred the real world myself. But that's interesting. So am I right in thinking then, as you described augmented reality, like a flight simulator? Is that the earliest, an early example of what augmented reality was all about?

Well, flight simulator can be done in virtual or immersive, but yeah, I mean, The very first augmented reality use cases that I'm aware of were the US Air Force actually providing see-through headsets for pilots and, you know, basically allowing things like, you know, showing them, you know, the level and pitch in a heads-up display, also targeting capabilities in a heads-up display and things of that nature.

Mm-hmm.

And so they could look down and see the, physically see the control panel of their aircraft, but then they look up and they could see other data overlaid on top of their field of view. And so that was some of the very first uses of augmented reality.

Yeah, that's interesting. When I think about virtual reality, then can you talk to some of the most logical applications of that in manufacturing today?

Well, as I mentioned, I mean, I think, I think training is one of the key areas for virtual reality. Right.

I was thinking augmented, excuse me.

Oh, I'm sorry. Yeah, for augmented reality, that opens up the doors to a lot more interesting capabilities because with augmented reality, you can overlay instructions on top of the real-world equipment. So that can enable for guided work instructions, for example.

Mm-hmm.

Which is actually something that our company does with our solutions as we enable that authoring and the execution of work instructions overlaid on top of physical equipment. That can also extend to live remote assistance, you know, calling an expert who can essentially then see through the eyes of that operator who's standing in front of the equipment and even interact with the equipment. you know, as they're working on it. Augmented reality, there are even some implementations in manufacturing that don't use a headset, but they actually use a projector and they overlay the step-by-step capabilities on top of the manufacturing line. So step 1 might highlight one area of the, you know, the bench or the workstation to focus on for that first step, and then the projector will point them to a different area if they need to select parts or components or focus on a different area of a jig or something like that. So a couple of different really interesting implications for AR in the manufacturing environment.

The projector solution would seem to me on the surface, just reacting to what you said, to be real interesting because I think there are some folks who would have trouble wearing glasses or might get nauseous from that. Am I thinking about that right?

Again, the nauseous feeling tends to happen for more immersive experiences where more of the real world is replaced by virtual objects.

Mm-hmm.

So the more that people's visual system is engaged, you know, like through the see-through glasses, then the less we see that discomfort that happens. But, you know, it really depends on the use cases. If you have a very standard, you know, like it's an assembly line, it's a very prescriptive, you're doing the same thing over and over again, then like the overhead, like projection-based AR systems tend to be, you know, easy to use and deploy, and you don't have any issues with people having to wear equipment. But if you have more sophisticated, highly technical skills, let's say it's like, you know, equipment operations or repair and troubleshooting, or, you know, some inspections and things like that, then you're likely going to have somebody wear a device. Or augmented reality can also happen, as I mentioned, like through mobile devices, whether it be a tablet or a mobile device. And so, you know, just by holding the device up, and using the camera on the device just to both, you know, see the world and also, like in the case of iPads and iPhones and even Android devices, many of them have built-in LiDAR capabilities. They can— they enable you to interact with the real-world equipment and provide a visual overlay. So things like inspections before you're going to, you know, you switched over a manufacturing line and now you're going to do something new. can be very useful for tools like that where a projection system may not be adequate.

That makes a lot of sense. You know, I'm thinking back to my days in operations and we would have, we would have a line expert, if you will, work on day shift Monday to Friday, but we ran 24/7. And so if the line went down and the folks at the troubleshooters on crew couldn't fix it, then he was always called, he or she was always called in to To deal with it. And I could just imagine a couple things that, that happen now with augmented reality. One is he can— someone could put a pair of glasses on or whatever and show him what's going on, and he can give them simple advice because normally it's something, you know, that someone's not seeing.

Yeah.

And 2, it, it really helps us. I can imagine it would help us, um, with trading. In other words, I don't have to— you know, my, my experience goes up really fast. I can get on the line And call for help more easily and, you know, be much more effective more quickly for 90% of the issues. You know, it's always that last 10% that, that usually takes experience and time to, to learn where we say, oh, you're not quite trained yet. So this augmented reality is really cool.

Yeah, that's where the augmented work instructions really come in, because if it's, uh, if it's your most common issues or your standard operating procedures, Those can actually be, you know, pre-recorded, ready to go. So somebody puts on the device and they, they just, they just follow the step-by-step instructions without any, um, you know, without any assistance. And like you said, for that extra 10% where it's something that doesn't— isn't addressed by one of those SOPs or common issues, then they can dial the expert. But the expert doesn't have to show up in person. They can actually, you know, remotely view and instruct the on-premise individual and guide them along in— that saves a ton of money in terms of travel cost and time. And, you know, we saw this really emerge during COVID when people were banned from traveling, and frankly, you couldn't have as many people in the workplace working next to each other.

Right.

And so that really kind of expanded the use of this capability within manufacturing.

Are you also seeing— so, you know, my background, I've got some industrial engineering background, and, you know, my mind's going along with, you know, what if I gave the individual the takt time in their glasses so they knew how, you know, how much time they had, if they were staying on pace or not? Are you seeing that sort of application as well?

Yeah, we've seen that in manufacturing. Again, that comes down to different use cases. So as you're mentioning, like if it's somebody who's really on that kind of assembly line process or they're at a workbench and they have a process that they need to perform, then you can have things like the takt time overlaid and they can— so not only do they have kind of step-by-step guidance if they need it, but then they can also kind of view where they're at with regard to metrics that that they're being tracked against or that the organization is tracking against.

Well, or that they want to just know how they're doing themselves. I mean, that's really the point is feedback is wonderful when it, you know, if I can, if I know what the standards are, and it's easy to follow, I can follow them. You know, a lot of folks don't, don't find that too difficult, you know.

Yeah, I mean, one of the other, the areas where we see a lot of benefit in the use of this technology is also reduction of errors, particularly for less skilled or less experienced workers and enabling them to— we have a manufacturer in the Midwest who they were able to take their in-class training time down from 3 weeks to 3 days because they were able to get people out on the shop floor sooner. And then those folks were able to— operate like a lathe, for example, with minimal assistance and error-free. And the management had confidence that, you know, with the use of these tools, that they were able to accomplish their job successfully and with minimal risk to themselves or to the equipment.

Those are some great, you know, case studies or examples that you share. What are some of the latest evolutions that's going on in the marketplace relative to both virtual and augmented reality that our listeners might be interested in hearing about?

Yeah, well, one area that I think is interesting is just the proliferation of kind of new devices. And I mentioned how, like, now the virtual reality headset players are getting into the augmented reality space. What they're doing is they're putting low-latency color cameras on their virtual reality devices, and they're providing— they're passing through the real world into the virtual world. Now, I would argue from a safety perspective that that's probably not the device that you would use out on the manufacturing line.

Mm-hmm.

But certainly you could take them and put them in front of, you know, a piece of training equipment, and rather than having a full digital version of that, they can actually interact with the, you know, with the real equipment hands-on, and they can see their interactions. And so it does amp up kind of the realism of that experience, and it permits a lower-cost device to be used in some of those environments. And then use cases like collaboration—

That's interesting.

Or remote assistance also are enabled by that. So, you know, that's an interesting advancement. what it's doing is it's also creating more competition in that space across the board, across both the AR and VR spectrum. And so we're seeing a blurring of the lines. We're also seeing some of the AR device manufacturers create more devices that can go more to the immersive side of the house. So Magic Leap, for example, on the Magic Leap 2, they have a capability that enables what they call global dimming. And so they actually, within their optics, they have a way to dim the experience so they can turn what's typically a see-through AR device into a more virtual device, kind of on demand, more immersive.

Wow.

So that's an interesting area. And again, I love seeing that competition because it just means that for, for companies who are looking into this technology and want to buy this technology. It's getting smarter, it's getting more capable, and it's also getting less expensive as time goes on.

Well, you know, I'm thinking about one of the things that we often talk about are how to get small and mid-sized companies started on their Industry 4.0 journey and how to help them do that. So if I'm one of those smaller and mid-sized companies who don't have a lot of internal resources. What's the best way for them to get started, for them to figure out how to maybe— or where should they start in this area of virtual and augmented reality?

Well, I would say, one, I don't think people should wait. I think that the point in time when all of a sudden your competition are doing it or your customers are asking for it, you might be late on the, on the curve there by the time you get to that point because, you know, we see companies deploying this capability at scale now.

Mm-hmm.

You know, this is— this technology has really turned a corner and it's maturing and it's maturing quite rapidly and especially with things like device proliferation. And another, before I directly answer your question, I'll just go back to the prior one. You know, the other thing that's happening is we're getting more ubiquitous, you know, wireless data through technologies like 5G.

Yes.

So, you know, there's both the public 5G, which people kind of hear about a lot when they, you know, look at the commercials during their football games about, you know, the latest phone that they might want to buy with 5G enabled, but private 5G, and those technologies are actually really expanding capabilities and enabling the ability to implement this technology within the manufacturing workplace without having to put a whole ton of infrastructure in place to make it happen. So that's another enabling technology, and from— in terms of getting started, There are a lot of, you know, you don't have to go out and buy a $3,000 to $5,000 headset to get started. A lot of the VR headsets are, I mean, some of these devices that I'm referring to are less than $500 to get started with. So from a training perspective, even with some of the pass-through AR capabilities, you can get started at the device level. Mm-hmm. Pretty inexpensively. There are also applications out there that you can run on tablet and mobile that will allow you to capture step-by-step procedures and including augmented work instructions and digital work instructions for, you know, really cheaply or in some cases free. You know, we on the App Store, for example, we have a tool called Manifest Maker. It's a free tool that small-medium businesses can go in and they can capture their augmented work instructions with pictures and videos and things like that. And then they can enhance them later with, you know, more augmented reality capabilities as they grow into it.

So I buy a device, I've got to buy some support as well, right? I've got to either be taught how to use it or I need someone to help me use it. How does that look normally? What's the best approach there?

Yeah, I mean, it really depends on your resource availability and how ambitious they are. There are plenty of tools online where people can learn how to do this themselves, you know, so community-based resources that folks can go to to learn how to create their own augmented reality work instructions. I mean, even, you know, companies such as Unity, who I'll actually have a representative from Unity on my panel at IME West, they have a platform that people can use to program their own AR and VR experiences.

Right.

Now, that's really sophisticated, but they have a bunch of free online tools, and they have, you know, tools that are free for individuals if they want to learn how to do that. Now, that's one approach. You have, like I say, you have to have ambitious, you know, smart, capable people that have time to do that, or you can hire a third party to guide you. There's many value-added resellers, systems integrators that kind of specialize in this area. We partner with a number of those, you know, for large companies, you know, large organizations, you could look to, you know, the Accentures, of the world, and there's, you know, we work with smaller VARs that also, you know, address, like, we have a partner, Lifecycle Engineering, LCE, who—

Mm-hmm.

They address manufacturing organizations, and they will, they'll guide them on this, and then there's common off-the-shelf software that folks can buy to implement on their own. So you really kind of have a You know, build your own, go to a system integrator or a BAR, or use common off-the-shelf software from a company like ourselves to implement it yourself.

Well, that's really great. It sounds like it's not so hard to get started. I just got to make the decision, huh, as a company?

Yeah. And again, you know, I would say that, you know, it's important. One, I wouldn't wait, but two, it's also important to to, you know, have a champion, but then also have, you know, somebody whose goals and objectives are to, you know, to implement this. Either— I guess you would probably start with an evaluation, make sure that you identify the use cases that are going to be impactful and meaningful.

Right, right.

And then when you go to implementation, just make sure that you've you've allocated resources to and given them enough leeway and time to do a proper evaluation and implementation. I find often what happens is folks want to explore it, but then they don't necessarily give people ample time or latitude to go and explore it. And so it's kind of, hey, check this out, but there's no serious effort to see how it could actually positively impact their business.

Yes, one of the things that I've heard said a lot is you really kind of almost need to have a small lab or something like that where you can tinker a little bit or allow people to tinker a little bit to learn what the capabilities are inside your own company, you know.

Is it— Yeah, or just even a proof of concept that runs over a certain period of time, you know, it doesn't have to be super long. I mean, I find 4 to 6 weeks is typically plenty of time to, to run an impactful evaluation, as long as you've allocated appropriate resources and equipment. You have to realize that folks will need access to equipment to be able to test and evaluate it. And so those are some of the considerations, and we'll probably talk about those considerations on the panel discussion as well.

Yeah, thank you. Well, that's great. You know, so as we kind of think about— you mentioned the panel, um, um, IME West is coming up February 7th through the 9th in Anaheim, and I know I'm really excited to be there. It's the first time I'll have been at it. You, you've been to it before.

What are you looking forward to?

Your third year. What are you looking forward to this year? Anything in particular?

Um, well, you know, it's always great to go there and see some of the, the latest, uh, equipment and, um, you know, things that they're showcasing on the exhibit floor. You know, 2 years back when I went, it was the middle of COVID and there weren't a ton of folks there, right?

Right.

So a lot of, you know, mostly exhibitors. There were some attendees, but last year when I went, a whole different story, you know, lots of attendees. I'm looking forward to just seeing that continue. I expect that there will be a lot of folks there this year. We'll be able to reconnect with with people we haven't seen in a few years. And, um, uh, so that's really, I think, what I'm looking forward to with this event.

Yeah, fantastic. It should be a really good event this year. I know there, IME West is talking about, you know, it's billed as the largest event in the country, so I'm anxious to see what that's like, you know. So that'll be good. And I think we're kind of reaching the end of our time together this afternoon. Do you have any final thoughts about, uh, augmented reality, virtual reality, uh, for our audience in preparation for the You know, for their own journey, if you will?

Well, I would say, I mean, there's plenty of resources to look into online. And as I mentioned, I mean, I don't— the time to look into this is now. Don't wait. You know, I really— not only am I passionate about it, but as I look at the advances in what's happening in industry, You know, whether it's across industries, you know, we do a lot in defense and manufacturing and, you know, even in the medical pharma space. But in manufacturing and defense, we're seeing this, you know, take off at this point in time. And so I think it's a good time if folks haven't looked at it to start to do their research, and hopefully they'll attend the panel and learn some more about it at IMU West.

Kelly, that's great. Fantastic. Well, I want to thank you for joining us today. I know you've got a really busy schedule, so I really appreciate you taking some time out just to share a little bit about these technologies with our audience. The Industry 4.0 Club hosts these kind of conversations on Tuesdays, typically on Tuesdays and Wednesdays, talking about issues related to technology, people, process, and culture. If you like what you heard, please join our LinkedIn page. And visit our website at industry4.0club.com. We really hope we see everybody in Anaheim February 7th through the 9th. It's still not too late to register if you haven't already. You won't want to miss Kelly and his panel on the afternoon of February 8th. So thanks, Kelly, and we wish everybody a really good rest of the week and see you in Anaheim.

Thanks, Mike. Take care.