The Key to Mobile Testing (Hint: Analyze, Analyze, Analyze) with Simona Domazetoska

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast and today we'll be talking with Simona all about the future of mobile testing, AI accessibility and beyond. If you don't know, simona has over 5 years of experience working in the software quality assurance industry with a special focus on how organizations can leverage the latest mobile and AI test automation technologies to really help drive IT modernization initiatives at speed, which is really critical especially if you work in enterprise. How do you scale especially when you have all mobile type of applications? Media test. We got you covered in this episode. Simona spends a lot of time crafting messages, go to market strategies and product research for the Tricentis platform. So she has a lot of knowledge with a lot of companies doing a lot of things, especially around mobile. So if you want to learn more about the future mobile testing, you don't want to miss this episode. Check it out.

This episode of the Test Guild Automation Podcast is sponsored by the Test Guild. Test Guild offers amazing partnership plans that cater to your brand awareness and lead generation and thought leadership goals. Get your products and services in front of your ideal target audience. Our satisfied clients rave about the results they've seen from partnering with us, from boosted event attendance to impressive roi. Visit our website and let's talk about how Test Guild can take your brand to the next level. Head on over to test guild.info and let's talk.

Hey Simona, welcome to the Guild.

Hey Joe, it's a pleasure to be here. Huge fan of your show. Thanks for having me.

I appreciate it. Really excited to have you, especially to talk about mobile testing. But before we get into it, is there anything in your bio that I missed that you want the Guild to know more about?

I think you mentioned it quite well. Thank you so much for that lovely introduction. So yeah, I've been working at trust centers for about five years, time flies. And yeah, it's. I'm in the quality space for quite a while, so thanks again for the intro.

Absolutely. So you seem to have a specialty around mobile testing. Why mobile testing? Is that something you focus in on exclusively or is it just something one of your passion areas?

It's something that we've been focusing a lot at Tricentis and mobile testing is a really critical Initiative. So part of what I do at Tricentis looks at how businesses are accelerating their digital business initiatives. So mobile, and particularly those that adopt a mobile first strategy, is critical. So that's certainly been fascinating to me and I've learned a lot through the project that we've been working on at Tricentre. So I think mobile is not just about software, it's about devices. And that goes into many different areas that I guess we'll be talking about. But that's just basically. Yeah. Why, why I'm into this topic.

Love it. Love it. So you mentioned one key difference. It's not just about the software, it's also about devices. So, you know, in general, how would you say mobile test automation differs from maybe software testing that most people may be more familiar with?

Yeah, so it's when you're testing mobile applications, you're not just testing the software, but you're testing how the software interacts with the hardware, with the actual device. And there's lots of different things that can be happening within the device. So if you think about it, you could be switching from a 3G to a 4G network, you could be logging into a public Wi Fi, you could be receiving SMS notifications, the device might have low battery, you could be switching phone carriers. And so, and not just that, but of course, users are in many different locations. They could be at an airport, at an underground metro station, you could be at a large shopping center or a football stadium, or even at a remote oil refinery. So I think that's precisely the challenge. How do you ensure that your mobile applications are working, well, high speed, in all these very complex dynamic scenarios that any number of users could be in.

Love it. So I know you speak with a lot of customers, like you mentioned, you notice a lot of the companies going through some sort of business acceleration. How many of them are aware of the challenges of mobile art that they actually need? Maybe a mobile testing strategy? Is it common now or is it not as common as. As you would think?

Yeah, it's a really good question, Joe. It really depends on the organization and the size of the company and the industry they're in. So you got to break it up a bit and ask, for instance, how mission critical is mobile to the face of their business. So typically, the more consumer focused and the more paramount the role that digital experience plays for the organization, the higher likelihood that they have already quite a mature mobile testing strategy in place. So if you think of retail or telecommunications or banking or public sector or even energy and utilities, any type of industry that's working on kind of capturing customers and ensuring that their customers have a friction free experience will already have some type of mobile testing in place. Having said that though, some companies are less mature in their practices so they might be testing manually. We have seen companies that set up their own device labs so they purchase a lot of those phones in house and they set up their own device labs. So while that's great and maybe can cut you some resources short term, what ends up happening is that teams spend up more time maintaining those devices. And we actually did a study, it can cost up to 200 to 400k a year just to maintain devices because they overheat. You need to ensure that there's security standards in place, employees can leave. So there's really a lot of glitches with that approach. So if you really want to scale, you have to leverage real device cloud. Having said that, you know, there are organizations, more mature organizations that have adopted a real device cloud. But then there's another challenge, right? And that challenge lies with understanding what exactly went wrong in the mobile application. And there's a lack of tools that really provide kind of analytics and you know, very detailed metrics to understand what exactly went wrong. We think it's all about test, test, test but actually it's all about analyze, analyze, analyze and fix quickly, ship it quickly. And you know, and that's what I think. You know there's so many things that could be going wrong and that's one challenge that we see with mobile testing today.

So what are the analyzing piece then most people think miss out on and there any key metrics is like the battery life of a device or like what are those type of key things people probably don't have insights into currently?

Yeah, it's there is an explosion of different variables and we like to group them into different types. So you could have performance issues for instance, low page load animation issues. It could be things like low frame rate or. And then you have UX related issues like blockiness, blurriness, colorfulness. How does the application actually look like? What is the background background look like? Then you have network and connectivity issues which looks at different wi fi's and connections and geolocation coverage and then finally device related issues which is what you mentioned like battery drainage or CPU usage or memory usage and how the mobile application may be impacted by any one of those different factors. Of course there's many, I can't list them all but that's really, you know, and not just that but I have to say There are sometimes even custom KPIs that are specific for certain, certain industries. Like, you know, if you look at shopping, online shopping, you know, from booking to shopping cart to purchasing. Right. That whole process, like how does that whole end to end process look like? Or if you look at, you know, making a flight reservation online, so flight to payment. Right. So different industries need different types of metrics and I guess that's sometimes the complexity with not only so many different mobile phones, but different types of variables that we need to look into.

So if someone's running on a device cloud then and a test fails, it sounds like it's more than it could be. More than just like it couldn't find a locator. It could be one of some of these numerous other things that you mentioned like performance, I guess, or you know, screen refresh isn't working. So people missing out on that insight then even though if they may be using a device cloud then and a test fails, they still have to do research and they may take a long time because they don't have these type of analytics available.

Yes, that's true. I mean there's depending on the different the real device provider that they're with and the breadth of and comprehensive metrics that they provide. But I guess that's precisely the challenge is that not all providers with of real device clouds have the all those complex metrics available. And sometimes it comes up to the developer to really work more time on debugging. In fact, I think there is one study that says that, you know, developers spend about 65% of their time debugging and that's not really the best use of time. So of course this is where we can leverage AI and machine learning to understand. And what AI and machine learning can do is aggregate certain issues that are found and display them as kind of issue cards so that, you know, developers can go in the tool and see where exactly it went wrong. How does that work behind the hood? Well, you have very often some type of AI framework or machine learning network that has studied the different types of issues that can arise with mobile applications. They study user flows and they aggregate that data and provide it back within the analytics platform so that, you know, developers can really tap into those and understand, you know, where the crash happened or where what was the performance over time. Is there any trend analysis like how does the application differ between different builds or between different networks or geographies? And that's precisely what.

Not all real device clouds offer that currently.

Absolutely. So also it sounds like, would this be able to help developers learn what they may have done, how they could have optimized their code so that they could not ignore it, but avoid these issues in the future.

Yes, exactly. And that's what it's all about. It's about improving.

Right.

And you know, I think there are some cases where obviously developers, when they're writing code. I think there was a study. I'm pulling out these studies, Joe, as they come.

There is a study that says that per 1,000 lines of code, you know, a developer may produce around 70 bugs. You know, I'm not trying to incriminate developers here. It's just that errors occur. And so understanding how that. What that led to is not always easy because then you have to go back to fixing that defect and it takes about 30 times longer just to analyze the code rather than to have some type of analytics platform that can help you intuitively understand that.

Absolutely. Also, I would think a lot of these things are hard to test for. Like how do you test 3G versus 4G or different performance things based on what network you're on? So I guess a device cloud should have these types of features as well. Does it make it easy? Does it let you know what type of test you need to run? Does it run it automatically for you? I guess the question how does a tester know what to test with mobile in a device cloud?

Yeah, I mean, it really depends on how you define your user stories and requirements and what's important to the business. I think that's a very separate approach. But going back to your network question, how would you test the network? There's typically two different ways you can leverage a real device cloud. So you test the actual mobile app on that real device, which is physically sitting in that geography using a certain mobile network provider, but using a certain type of mobile network speed. So that's usually the more, I would say, expensive solution. Right. But definitely the most accurate. You can then also leverage a real device cloud to simulate that network as well. So typically what happens here is that you have the real device connected to some type of WI fi, which is then connected to a router, and then you have some type of network shaping that software attached to the router. Another option, and probably the most cost effective and fastest way, is to leverage a virtual device where you actually. It's not a real device, it's a virtualized device. It's a copycat of the mobile device. It's not the real device where you can actually simulate the network. But then that's not great because you might not have realistic or accurate results. So those are some ways that you can approach that.

Nice. I guess you get into that debate then. Should I use a simulator, an emulator versus a real device? Are there any key differences then when you should use one over the other? I know you gave a few just now, but are there any other ones that someone's trying to decide, okay, should I use an emulator, a simulator or a real device? It does even matter nowadays?

Yeah, it's a really good question. And again, I would say it really depends on the organization, it depends on the team, it depends on their budget. How mission critical is the mobile app to the face of the business? If you're talking to an online fashion retailer who you know they're trying to capture a large audience, the mobile app has to be friction free. It has to be seamless to work top notch. Right. So I would say it's really important to combine both. You know, you need to combine real device, cloud and virtualized devices, otherwise called simulators and emulators. So when do you use which one for? For what scenario? I would say a general rule of thumb is if you're trying to look for a cost effective, highly scalable solution, something that's really quick and easy to set up, and also where you just want to check the functionality of the app in the early stages of development, go for a virtual device cloud. But you don't see customers with virtualized devices in their pockets. Right. You see them with the actual devices. So that's why it's also critical to if you want to have test all those real user scenarios that we spoke about earlier, you need to leverage a real device cloud. So of course it's more costly, but in terms of the different scenarios that you want to cover, you need to leverage it.

Absolutely. So you mentioned one of the keywords or the buzzwords of the year, AI earlier, but we didn't really dive into it. I want to pull it out now. So I hear AI applied to a lot of things in automation. I'm just curious to know how you see trends in mobile testing as it applies to maybe AI and mobile. Is there anything specific you see as a trend that AI could really help with mobile? You mentioned some analyzing your data. Is there any other features or things that could help with.

Yeah, there's so many different things and I'm sure you've heard some of these. Right? You're in the testing space a lot yourself. Visual testing is really critical that obviously looks at, looking at finding cosmetic bugs between different versions of the application. Not just cosmetic bugs, but how does the application look? Is it easy to access? Is it easy to navigate? Is it accessible? Which leads me to the next topic, which is accessibility testing. So obviously, you know, organizations want to cater to different audiences. This is a topic that's actually quite close to my heart because I used to work in the humanitarian sector. And so I think, like, if you look at, we tend to think, okay, this only applies to people who may be blind or have hearing difficulties, but actually anybody could be experiencing some form of visual difficulty. Right? You could be an a bit older or have to wear glasses or whatever. You know, there's so many different complexities. So testing for accessibility is critical. But going back to AI, like I think a performance analysis is really critical. So there's. You can leverage machine learning to predict and diagnose performance, performance issues across applications. That's one. You can do things like crash analysis, where machine learning is employed to look at vast amounts of big data and identify patterns across the different crashes. You can do trends analysis, as we mentioned earlier, right. Where you're looking at, you know, different runs across different devices and network conditions. And then also machine learning is used to detect glitches and anomalies and video and audio quality. So you have, you would test the machine learning systems to identify those issues and then spit that data back to the developer to identify what went wrong.

Yeah, that's a question I get asked all the time, how do I test video or audio? So it sounds like using this mobile AI embedded analytics tool, it would be able to bubble up those type of potential issues.

Yeah, correct. It is definitely possible to do that. Right, Very cool.

So you know, a lot of times you mentioned accessibility testing once again, is there anything you see or know of that people can do to make their mobile devices more accessible before it actually goes to production going, oh boy, it's not, not accessible at all.

That's a really fantastic question. And typically I find people are becoming a lot more aware about successibility in this topic. And what that means is before you end up building the final version of your application and then you're done and off with it, developers are taking those accessibility standards and putting them early into their software development process. So that could mean things like embedding ARIA labels within the code. And ARIA labels, as we know, are used by visual assistive technologies like screen readers to identify different objects on a page where they actually call out those different objects. Like this is a table, a scroll or enter or whatever. So embedding, having that coding within the application is certainly critical. Not just that, but there's also many different manual. You can even do manual testing for accessibility, like looking at application visually, you know, does the color make sense? What about people with color deficiencies or issues? Are there certain colors that are critical for the application that we may need to consider changing? If it's red for instance, or green or so forth. Right. So you can do all those, implement all those user experience standards early on in the application. By the way, don't just take my word for it, there is a. The WCAG guidelines are available online. There's a whole community, you know, different accessibility experts and standards that you can. And guidelines that you can adopt for, you know, people wanting to learn more about it.

Nice. There's also usually a debate of should I use native frameworks to test my app or should I use like higher level all purpose, like something like Appium to test my apps or something else? Where do you stand on that? Or what do you see customers using or what would your approach be typically?

Yeah, it's a really philosophical question.

Yeah, it really depends on who is testing again and how is it being leveraged by the business. And so if you implement a, say you're using a codeless test automation tool, obviously the advantages there are that different Personas within a large organization can contribute towards testing different lines of business. Personas that may not have knowledge of scripting. And also it's not just about scripting knowledge, it's also about maintaining tests as, you know, the software updates. How do you maintain those in an easy fashion? Long term, once you build a regression portfolio over time and you have thousands of lines of script, it can become quickly difficult to understand what went wrong. And not just that, but again it goes to the breadth of testing. You know, are you doing end to end testing, are you doing unit testing or are you doing, you know, right, like component testing. So if you're doing a really complex end to end process that touches multiple apps, I don't think, you know, scripted approaches are best. But then again, if you're wanting to do unit testing, check code, I certainly think that scripted tools are better suited for this, you know, type of testing.

So one of the things I always say, one of the reasons why I love speaking with folks that work for vendors is you get to see a bunch of different scenarios, I'm sure a bunch of different customers. Do you notice like what the trend has been? Is there any trend? Who does the mobile testing nowadays? Is it developers? Is it testers? Does it depend?

Yeah, it's a really interesting question as well. You know, it swings, the pendulum swings between both, I would say. And again, it goes back to how testing is set up in the organization. Typically what we find is that larger organizations that have a kind of centralized testing center of excellence or strategy in place, it typically testing will be assigned to the testers who will implement testing. But then with quite decentralized organizations where testing lies within the agile teams or within the lines of business, it could be as well that it's the responsibility of the developer working side by side with the tester. I do think that when talking about the analytics components that we were speaking about earlier, that could be useful for both developers and testers. You want developers to be focusing on building the application, but they should absolutely be aware about the quality. So that's where they can leverage those analytics platforms and collaborate with testers real time to improve the way that they code. You know, not just that, but really have insight into the application from a business standpoint. Yeah.

Great. So I do also have a news show podcast and I've noticed that, you know, Tricentis has been acquiring a lot of cool tech lately. A few years ago now I think acquired testim and I think recently it acquired a solution called waldo. Just curious to know, how does that all work? How does that help expand what I think you already have? You have a Tricentis device cloud. Maybe we could talk a little bit about know we've talked about all these pain points. Maybe we could talk a little bit about the solution like Tricentis device cloud and how all these acquisitions, how are they coming together to help test or a lot of these really tough things that people are struggling with.

Okay. I can tell you that the strategy at Tricentis is to provide a really comprehensive mobile testing platform and that looks at all aspects of mobile testing. Not just testing on real devices, but also on virtual devices. Not just having low code scripted approaches, but also having no code approaches. We're aiming to cater towards different sizes and segments of companies. Obviously we've been quite, you know, enterprise driven, but we also slowly focusing on the SMB space. So we're trying to build a platform that's very comprehensive because why is that important? And it goes back again to serving our customers because really it's all about our customers. We find that customers tend to embed both real device testing and virtual testing for different use cases. Right. So if it's something that, you know, if they want more, as we mentioned, cost effective, easy to scale solutions, they're more likely to embed virtual testing. And then if you want to test more deeply, get more accurate results, you will combine that with real device testing. That's what we're aiming for, to really have a holistic strategy in place. So that's really part of what we're doing with Waldo. Yeah. So I hope that covers it.

Yeah. In my notes I have something virtual SMB space. What is that?

So we find that, you know, small to mid sized companies might not always are more budget focused, so they might not have the resources in place to test across real device clouds, which tend to be a bit more costly. Having said that, they are also even able to test across real device clouds because obviously there's different types of models in place. You could have access to devices or dedicated devices through private clouds, otherwise called a single tenant option. That tends to be a bit costlier. But then you can also leverage a shared device pool. Right. So most real device providers out there offer a shared pool of devices that different organizations can access and that tends to be the costlier approach. So typically for smaller and mid sized companies, we do strongly recommend combining both testing across virtual devices, but also real devices and particularly leveraging the shared pool.

Nice. I just assume when we talk about mobile testing, a lot of people think about apps, But I know IoT has been out there for a while. Do you see people struggling with, okay, I have an IoT device, how do I test this thing? So not like, how do you handle IoT type devices, then type testing as well?

IoT is a very hot space and it goes hand in hand sometimes with 5G technology. Right. So 5G Networks is out, it's rolled out. Obviously we need to leverage high speed, low latency advantages of 5G and that goes hand in hand with IoT. So if you look at, for instance, the automotive industry, sometimes you have smart vehicles or you have this kind of in car experiences. So sometimes you may need to test how mobile interacts with these devices and these other hardware systems to tap into IoT. You could also have in manufacturing, you have smart factories. Right. So 5G supports IoT in looking at machinery and tracking how devices are communicating real time. So you could even have things like agriculture, precision farming. Right. Where you have drones and machinery all communicating together to understand fields and crops, real time. So I personally, I find this really fascinating space. There's just an endless amount of use cases. Right. Gaming, retail. If you look at retail, I mean, nowadays you don't even need to go to a store. You know, you could try the actual clothes online using Augmented reality applications which leverage 5G networks. So the use cases are enormous. And I think some of the top kind of more innovative companies are certainly adopting IoT to test them across different devices and networks. Yeah.

So AR VR, you just mentioned it when you're trying on clothes, I assume. Does this have anything to do with running something in the cloud as well? Is there real devices for AR VR as of yet or is that still something that maybe is a gap?

Yeah, that's a really good question. I think maybe there's something cooking in the kitchen.

Nice. Cool, cool, cool.

But yeah, I would be happy to hear what you would think about this, Joe.

No, I think it'd be great. I actually talked to a lot of developers. They have kids and they have their kids test it for them. So having a more scalable solution I think would be very, very helpful.

Definitely. Yeah. It's a very exciting field.

Absolutely.

So something that, that we're looking into.

Awesome. Okay, Simona, before we go, is there one piece of actual advice you can give to someone to help them with their mobile automation testing efforts or device cloud testing efforts? And what's the best way to find contact you all learn more about Tricentus solutions that actually help with the mobile testing space.

Sure. I would say put the customer in the centerpiece of what you do and really understand what is the customer doing, what are they trying to accomplish? And then don't just test, analyze, analyze, analyze and really understand what's going wrong. So that also requires communication. We shouldn't forget the human component of testing. Not just testing, but developing and beyond. And you know yourself, right? We have AI and machine learning. It's such a buzzword. It's like you have to talk about it. But we shouldn't forget the human component of what we do. So I think that's really critical in terms of how you can reach out to us. You can head over to our website, you can actually try Tricentis device cloud. We have a free trial available and then obviously you can contact us directly on our through our website. If you want to learn more, we can put you in contact with our product experts to provide those demos.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode, head on over to test guild.com.

And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'M Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the guild community, I'd love to hear from you. Head ON over to testguild.info and let's make it happen.