220: VR and AR in Mobile Apps Testing with Bj Aberle

Hit it.

Welcome to the Test Talks podcast, the place to go to geek out on software testing. And now your host whose mission is to help you succeed with test automation, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of Test Talks. With the ability to experience VR on mobile devices and web VR on the horizon, the potential for VR to go mainstream is huge. So what does that mean for testing? That's what you'll discover in this episode as we test talk with BJ Abele all about VR and AR in mobile apps testing. So get ready to discover how to deal with new test environments, configurations and interfaces for testing virtual reality and augmented reality mobile apps. Check it out. Test Talks is sponsored by the fantastic folks at SM DOS Labs, the cloud based automated testing platform that eliminates the need to maintain your own selenium grid and test infrastructure. Try it for free today. Visit testtalks.com and click on the sign up now link under the homepage Sponsor section.

Hey bj, welcome to Test Talks.

Hey Joe, thanks for having me. It truly is an honor to be part of your show. So thanks for having me.

Awesome. It's great to have you on the show today. If people don't know, you've been the composer of mostly most of the Automation Guild and Perf Guild soundtracks for our promotion. So thank you for that and today I noticed that you gave an awesome session recently at Star east on AR and VR with mobile apps and I thought that's what we talk about today. But before we get into it, could you just tell us a little bit more about yourself?

Sure, absolutely. My name is BJ Aberly. I am the QA lead for Float. We are a company based in central Illinois of all places and we work mostly with mobile solutions for B2B. So an enterprise. So trying to figure out how and company can take order fulfillment on and their sales force out in the field and integrate those to E commerce solutions. So we do that. We do a lot of E learning work for enterprise and then we do some government work as well. And then every once in a while and more frequently than not now we're getting solicited for computer vision, augmented reality and virtual reality projects.

Very cool. So I'll be honest, there's not many folks I know that actually doing AR VR especially testing for some reason it's still a manual process. So just curious to know how popular do you think this is becoming and is this a trend you see that'll be escalating shortly that people should be aware of?

Oh, I think. Well first of all, to answer your first question, yeah, it's in its infancy. It's one of those technologies that's been on the Gartner hype curve for a long time and now we're finally starting to see it kind of moving up towards the plane of productivity. And people are seeing that there's more uses for augmented reality, especially just kind of the Pokemon Go uses. So I think people, once they realize that you can use augmented reality for much more than just overlaying video on top of your camera view, then there's going to be a lot more uses for that. And I think people are starting to realize that.

So since it is fairly new, people are pretty used to testing. Especially good testers know how to test. But is there anything they need to adjust in their mindset or how they approach their test plans when dealing with an AR VR type app?

Well.

I think the model right now for testing AR and VR are three simple steps. The first one is freak out because no one else is doing this and there's probably no project that is like yours. So there's no model or recipe to go Google. There's no stack overflow to look at. You're just going to have to kind of roll your own for a while. Then once you get over the freak out portion of it, you can actually start testing. And the problem with testing, especially augmented reality, it's really kind of a blackest of black box testing you can think of because you're taking an input that you think the device should be getting and then you're trying to see if, well, is this really the correct output? And it's not until a matter of trial and error that you kind of get familiar with the app. And then once you do that, then you start working with your developers and then you kind of start coming up with some creative solutions to the testing problems. But it's kind of like this, this evolution of oh crap, then okay, and then finally it's, oh, cool. That's kind of what I talked about in my Star east presentation is like you're going to go through these through three phases. You may name them differently, but it's kind of like with anything at the beginning, when you know so little about it, you just kind of freak out. But it's the time that the discovery phase of educating yourself and learning about the technology and then actually providing some useful feedback, that's a very scary place to be in when you feel like you're not providing value, you're just kind of learning and reporting your observations. Back to the developers. So you're right, it's an evolution right now and it's very. The projects are just. It's in its infancy, the technology is in its infancy and so it's going to be a while before we have viable frameworks.

I think I know when I speak to VR developers themselves, it's usually with the desktop app and the whole headgear on. But maybe. I guess I'm just old. I have never used AR or VR on my mobile app itself. So do. How does that work? Do you need gear and if so, how do you test the gear aspect of it?

Yeah, so this is the Google Daydream goggles. Basically. You can see that they look like a regular kind of VR set, but your phone actually goes in between. Here, here's a controller for it. You have a controller, the phone. I'll grab my phone here. So the phone would then get sandwiched in like this. It would lock and then it just uses the accelerometer and the Imus from the. On the device and it displays a virtual reality environment. We actually have another cool technology here, which is called. This is the MIRA Prism. This works for augmented reality. And instead of having the phone in front of your eyes like this, it actually puts the device in the headset like this and it will reflect on the prism here. So it's kind of a mixed reality experience. And that's what. We're working on a project for that right now. So you wear the head, you wear it like this and then you can see not only the real world, but you can see the digital projection as well at the same time as a mixed reality experience. So those are a couple options for AR and VR for mobile.

That is so cool. I've never. I've never seen that before. So I'm just curious to know someone that wears glasses. Can they still wear those headsets or do you have to take them off? Does it still work if you're blind as a bat?

Yeah. I would say for these you probably need some sort of. I mean, there's enough space. I don't know. That's a good question. We all have to test that out for the AR ones. No, you can wear your glasses and be fine with that.

So do you. Is there a way you can record your session with interacting with the app and then so developers can watch it? Because it's such a visual type thing that if you find a visual bug, just thinking out loud is that. That's something that you would do?

Yep. So one of the things you can do, especially if you're developing on Daydream platform, which is Google Android platform is you can use Chromecast and that will actually project what you're seeing in the headset onto a smart tv. So you could record that way or you could have developers see what you're seeing as you're testing. There is another plugin and Google Instant Preview that you can use when you're doing Daydream development. What that does is it actually allows you to preview your build in Unity in the editor. So instead of having to put on the glasses, you can just run your game and you can hold the phone and tilt it around, but you can still. Everyone will be able to see what you're able to see in the editor. So that's another way to kind of get around that as well.

Awesome. And like I said, I've spoken to developers of virtual reality before. The way he tests it, he has his kid put on the headgear and then he has him do actions. He calls out actions and he's watching it. So it seems like a very manual process right now. I know it's still in its infancy. So do you see any tools coming out lately that maybe can help automate some of these more manual type interactions?

Well, I think one of the things that we're trying to do with our augmented reality projects is to figure out how we can do unit type testing. One of the things that we've been able to do is in Xcode, we'll be able to launch the app and able to actually inject a still image. One of the things that we had to do is we had to actually detect text.

We could put up the still image and maybe give it a few different variations, like a black and white version of it, a sepia version of it, and a blurry version of it and test to see if it's still interpreting the text correctly. So I think it's going to be small, but I think the unit tests are going to be really important going forward. But you're right, it is still just a big manual effort once we get builds going and we need to do some testing.

So with these type of devices, I'm sure over time there's going to be all these weird permutations. Any thoughts on how you're going to handle that?

Well, it's funny you mentioned that. I was imming Jonathan Lips a couple weeks ago and he was like, man, you should really think about coming up with some sort of Appium integration with that. And I was like, that seems a way above my pay grade, dude.

But he, I think just talking to him, he's already got ideas because then he kind of laid out sounds like, well, you need to do this and that and that. And you know, he's like, I know it sounds tough, it's a lot of work, but I was like, I'd be willing to talk to you about that. So I know Jonathan's probably thinking about those sorts of things and I just don't know. I don't know of anyone right now that's actively pursuing that.

As I was stalking you, I came across some of your talking points for your session at Star east. And one of them was you were going to talk about what data you need to test AR VR. So what data do you need to test AR VR?

Well, I think one of the key points for testing AR and VR, especially AR is that you have a physical environment now that you actually have to test within. So you think about like all the things that you have to do just to set up the tests. And we have a couple SAS projects that we work on. I think about like everything that I need to do as far as pre production just to get testing going. So like I have to have no database set up in the right state. I have to have the right environment, the right user, the right docker container, all, you know, right access controls, all that sort of stuff just to be ready to test now take that sort of precondition and now you have to have like a physical face that you're testing in. If you're doing ar, do you have, you have to have markers that need to be printed out. Not only do they need to be printed out, but they need to be printed out as if you're in the production environment. Well, what does the production environment look like? Are there a lot of windows? What's the ambient like in that environment?

In fact, we had a. One of the things that we did last year we did an app for a conference and the idea was like there was no written text or conference information on the walls, it was just AR markers. And you'd go and you'd go out to the conference room and you'd hold your phone up there and information would come up on your phone and about who was speaking, what they were talking about, all that sort of stuff. And I was like, before that I said, okay, what are the markers going to look like? What are they going to be printed on? All that sort of stuff. Like, okay, BJ, they are going to be on 8 by 11 pieces of paper and they're going to be like this. I was like, okay, great. So can we get, are they going to be in color, all that sort of stuff. So I got the markers printed out. I could test them here. But when they went to San Jose and there is actually in the environment, they failed to mention that they were going to laminate the markers. And so you can imagine I'm on Slack channel going like, I did not test this because this was not told to me that they were going to be laminated. So things like that, you think about going in like, what is the actual production environment looking like for AR Now?

Maybe I'm crazy, but it's almost like we're going back to before automation in the sense that as more things become more automated by AI and types of things like that, there's a lot of activities that cannot be automated regardless. And it sounds like maybe VRAR is going to be the same thing like this. It's almost like we're just going back. You just need to be a great tester. So maybe you should work on your testing skills because no matter what technology comes out, there's going to be things that a tester needs to think about. Like who would think about, oh, I need to test laminated, a laminated card or something. But that is something that a tester usually would think about as an edge case. So it's interesting.

Right? Well, I think you're exactly right. And I think those people who are fearing that AI is going to take over their testing jobs, I think you have the Gartner hype curve, right? If you're on the left side of that curve, if you can stay there, there's always going to be a need for manual testers. Right. Like that technology is just going to be so new that it's going to be hard to artificial intelligence take over everything. So I think you're right. It's just going to be manual testing. And the idea of being a good tester is going to be paramount. And the idea of automating these things is still going to bring efficiencies, but having that mindset of being a great tester is still going to be first and foremost as the skill to be desired.

I think another thing that I saw in your abstract for your starry talk was something to do with Google's Tango ship. Just curious to know what is. I've never heard of Google Tango. So what is it and how do you leverage it?

Yep. So there's a couple things about Google Tango. Google Tango was a hardware and software technology Stack that was introduced a few years ago by Google for Android devices. And what Tango allowed was for some extra capabilities for mobile devices. So there was a infrared camera, which was a depth sensing camera, a wide angle camera, along with the regular RGB camera. And what that allowed the device to do is it allowed to be what's called environmentally aware. So the device could scan a room and it would solve the problem of slam, which is simultaneous localization. And but basically it meant that the device could scan a room, record the key features of that room and know where it is in relation to that room. And if you walked away and you came back and it was still scanning, it could recognize those features and say, oh, I've been here before. Not only have I been here before, but I know where I am. So what Float did was we were able to take that technology and create an app for Tango enabled devices called Sedalion. And Sedalion is an app that allows blind and low vision users the ability to actually navigate unfamiliar environments in a more efficient way. It wouldn't replace like a cane or a guide dog, but it could tell them you could have. They'd wear the device like on a modified running vest. So it'd be pointing out this way and it would scan and if there was something on their right, there would be a little audio cue on the right ear, on the left would be a little audio cue on the left ear and it would say, hey, you know, basically it would be warning them you might want to turn or change your course of direction because there's an object in front of you. And the same with like a head level object, it would tell you, hey, you might want to be careful or slow down because there's something head height that you might bonk your head into or same with, same with a curve or a low level object. It would give you give those sorts of warnings. So we were able to take that technology and leverage it for, for helping the outcomes of blind and visually impaired people.

Well, it's very cool how you came up with that. So how did that come about? Did you tell your developers, hey, who came up with that idea and who helped guide them to say, this is a great idea to invest time in. I know it's not part of the feature, but it really would help with testing.

So one of our developers here actually presented one of our projects at, I think it's called the Vision Summit or something, is a conference for computer vision. And some people from Google were there and saw what we were doing and approached us to help them out with some technology with this project they were calling Tango at the time. And we were fortunate enough to partner with them for a little while and run with this idea that we had using Tango technology and to see what would come of it.

Very cool. So also, what I love when I speak to speakers like. Like yourself, when you gave a session at a conference, I know they have a Q and A after. And sometimes you may get a question. You're like, I didn't think of that, or I should have covered that. Did you get any of those types of questions that made you wish maybe you covered it, or maybe you haven't thought about it until that question was asked?

No, it was interesting. The first question that I asked at Star east was, who here has had an AR or VR experience? And no one raised their hand. And I was very surprised at that. There was one person that was testing VR for an auto manufacturer, but other than that, there was no one who had had even, like, put on a headset. And I thought that was really interesting. So I was like. I was like, well, I could say anything, and you guys be like, oh, great point, ej. You know, I was like, I found that very interesting.

So I'm really gonna age myself here, though. But I've had one experience, and it was in 1991 at a college in Uri in Rhode island, where they had this. This huge, huge device. You'd sit in, you put it on. I don't know if, you know, Dire Straits had a video called I Want My mtv, but it has all these block figures. That's exactly what it was like. So it's so funny that I actually have had a VR experience. So cool.

Well, it's funny. It's funny, too. I mentioned a technology called hyper reality, and in my talk, and Melissa Tandi actually had tweeted out she never heard of hyperreality before.

And after the Star East Conference, I went to Disney Springs, and they have a hyper reality experience, which I highly recommend to anyone. It was amazing. But the idea of what hyper reality is that it's virtual reality that's based in the real world. So right now, virtual reality, if you have a mobile device on, it's not very safe for you to walk around in virtual reality because it can't track you. But what hyperreality can do is it can actually make the world a different experience for you. So, for instance, the experience, the hyper reality experience that my wife and I had when we were in Orlando was we were Stormtroopers, and we had to, for Star wars. And we had to get some secret code and bring it back. But I could turn around and look at her, and she was a complete stormtrooper. And. And so was I. And we could. And it was just so amazing. Like, we. We would stand there and I could see a doorway in front of us, and I put my hand out and like, wow, it's an actual doorway. And I could actually walk through it. And we were in this one room and my wife was like, oh, look, there's guns or laser guns or whatever. And. And I thought they were just kind of like wall or men. And she went over and she picked one up and like, it was an actual model of a gun. Like, we were supposed to use that in the experience. And I was just like. Like, my mind was completely blown at the time. Like smokes. This is. This is amazing technology. And. And I got to experience hyper reality for the first time.

Wow, that is awesome. Now, where's this again?

It's at Disney Springs in Orlando, which is the south of Disney proper. And it's. I can't remember what it's called. It's called the Void. The Void is the name of the place. And I would imagine things like these sorts of gaming and virtual reality experiences will start popping up around the country pretty soon.

As you're testing AR VR applications, are there any pitfalls or common pitfalls you've seen and you think other people may run into that you've solved that maybe you can help them before they have to find a solution?

Sure. I think with the sodalion app, I spent a lot of time walking around and kind of not knowing if the results that I were getting, my observations were truly helpful or not. And I mentioned this in my talk is like, for a period of time, I just had to get familiar with doing the app's personality and attitude. And I think testers kind of know what that is. Like sometimes build. You get the build and you launch it. And for whatever reason, it just. You can tell that it's not working the way it was before. But I know that's not scientific, so I wasn't happy with that. So I started trying to get a lot of quantitative data to help support my observations. But ultimately what I did was I had to go to the developers. I didn't had to. I went to the developers and I said, look, I'm not happy with the way I'm testing this. How can we solve the problem that I'm having? What we did, what we came up with was this idea of that infrared camera. That point Cloud Camera is creating serialized data. Why couldn't we just launch the app and basically mock that data back into the app instead of actually having the Tango service create it? When we did that, we were actually to. It gave me a couple of things. I could repeat tests. I mean I could repeat and repeat and repeat if I needed to. But also I could set. This sounds so elementary. But I could test with the debugger attached. And when I could do that, I was like, oh my gosh, I'm seeing so much more into the app than I was before. And so now after that I could go out and I could start capturing a bunch of different scenarios as this point Cloud data or what they call the ADF area area definition file. I could have all these different scenarios that I could test. And once we started seeing major deviations that gave us a clue. It's like, okay, maybe there's something wrong with the algorithm now that we need to go look at. So I thought that was kind of like, I think there's opportunities to mock data. I think we just need to figure out like the SDK manufacturers and people at Unity and Unreal, they kind of have to, we kind of have to talk with them and kind of say, hey, this would really help out as a testing, as a web hook. Not a web hook, but as a test hook to come in and be able to do a few different things that I just think it's just not there yet.

So bj with the Tango technology, just curious to know how does that line up with what Apple is doing with their front facing camera?

So basically the technologies are the same. One obviously is the front facing camera and one for the Google products are a rear facing camera. I kind of liken it to a potluck that was being held and the potluck started at 6 o' clock and tango technology showed up at 5:30 and they brought shish kebabs. So like these awesome skewers with vegetables and meat and really good looking, really good looking stuff. Apple decides to show up to the, to this dinner at 9:00 and not only do they show up at 9 o', clock, but they say, you know what, this party is vegetarian only. So your shish kebabs are gonna have to go. And we're, and now the party can start. But the whole time Apple is saying that everyone at the party is going like, man, that meat looks pretty good. So what they've done is they've taken that instead of like where Google is trying to like take in all this information from the world. Apple kind of flipped it around. Well, literally flipped it around. Like we're going to do it small and we're going to do like face detection and we're going to like very simple things and tease out this technology and kind of do this on a smaller scale. Where Google tried to, I think, boil the ocean too quickly and you only had really two manufacturers that created Tango enabled devices.

Okay bj, before we go, is there one piece of actionable advice you can give someone to help them with their AR VR testing efforts and let us know the best way to find or contact you?

Sure. So the first part, I'm on Twitter at Bjeverly, so I'd be the first place just DM me or reach out to me that way. As far as a piece of advice, I think we as testers are in an awesome position to align ourselves with the development team because AR&VR is so new that sometimes the developers working on these projects, this might be the first time that they're working with these APIs. That's a great time for us to educate ourselves as well. So my advice would be go to Unreal or Unity and just create a very simple AR or VR experience. I would say with the audience that you have here, it would be pretty simple. I'd say within a couple hours you'd be able to have something up and running. And what that does, it gives you perspective and context as to how these apps are created and it shows you like, oh well, the Marker database actually can reside in the cloud or it can reside locally on the device itself. And that gives you some opportunities. Like now I can actually mock some data.

I can set up the device to be testable a lot quicker without asking as many questions. And I have some understanding. I'm going to take a. I'm going to do a second piece of advice too. At the beginning of the year, I made a concerted effort to be more integrated with the developers. In fact, I moved my desk into the development queue and that has made a huge difference to me to be able to turn around and say, hey, I'm working on something, but I'm not quite sure I understand what's going on, what's going on. And they can come over and within a couple minutes we can talk about a problem or surface an issue and, and work on a way to resolve it pretty quickly. So that's my advice.

Thank you BJ for your AR VR testing awesomeness. The links to everything of value we covered in this episode head on over to testtalks.com 220 and while you're there, make sure to click on the Sign up for a free trial link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And also check out some information around our new Guild conference happening February 4th to the 6th. Automation Guild 2019 is back, the third annual online conference dedicated 100% to helping you succeed with Creating Automation Awesomeness and as far as I know, the largest automation conference in the world that's online. If you want to learn more, just make sure to check that out@testtalks.com so that's it for this episode of Test Talks. I'm Joe and my mission is to help you succeed with creating Automation Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Talks podcast. Head on over to www.testtalks.com for full show notes, amazing blog articles, and other automation awesome this.