Using AI in Mobile Automation Testing with Shannon Lee

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host Joe Calantonio.

Hey it's Joe and welcome to another episode of the Test Guild Automation Podcast. Really excited about today's podcast because we'll be talking with Shannon Lee all about using AI and mobile automation, testing and all other types of automation topics. Shannon Lee is in Atlanta, Georgia native working as a developer evangelist for Kobiton to help bring the good word of innovative technology to the world. So really excited to have her. Shannon, if you don't know, is an avid learner who enjoys challenging herself and learning new trends in software development, especially test automation and applied data science. So she really knows about AI, she really knows about automation and she knows about development. So we got the best of everything. Given her positive and collaborative personality. If you know her, Shannon really enjoys engaging with like minded individuals with goals of inspiring encouraging others, especially women in technology, which we need more of. So thank you Shannon for being a great example as well as learning from for brilliant minds within the DevOps community. Really excited to have Shannon on the show with us today. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSourceLabs and click on the exclusive Sponsor section to try it for free today. Check it out.

Hey Shannon, welcome to the Guild.

Hey Joe, thank you so much.

Awesome. Great to have you. It's been I met you earlier on a earlier demo of Kobiton, so really excited to have you on the show to talk more about AI and automation, your views on development as well. So Shannon, before we get into those, anything I missed in your bio that you want the Guild to know more about?

No, I think that was pretty spot on. So yeah, again really just love talking tech and really excited to be here with you and doing just that. So thank you again for having me.

Awesome. So you are a hardcore developer so I just curious, I'm already off script what got you into automation testing?

So I actually my career started in testing so I was a manual tester first working as I was working through software development and one thing that really kind of engaged me is the the ability to automate your manual test cases. I thought that was something really cool and was really fun to watch and kind of dive into. Also kind of, you know, seeing your development and kind of kicking it off and seeing it happen in real time. So it was a lot of fun. So it really, really got me interest. Something about automation though is that it has its drawbacks, so to speak. So really excited to speak with you today about some of the tools and solutions that we have that can kind of help combat those drawbacks so that you can really make use of all your time in your automation effort with that. So great.

Yeah. So I love it. You have real world experience. So you know what people are struggling with in different. The techniques and tools we could probably use to get around it.

Absolutely.

Nice. So, you know, a lot of people I speak with, they're very familiar with web browser automation. Mobile application automation has been around for a while, but still I see teams struggling with it. So I guess before we get into it, how do you define what is mobile application automation?

Sure. So first of all, mobile application automation is very, it is difficult to achieve and there's, there's a couple of different reasons why that, why that is so for why with mobile app automation you really have to take into consideration the different platforms that are in play that kind of set the tone for how their applications are developed and how you kind of have to follow, so to speak, their rules of the road, if that makes sense. So if you look at iOS applications, you know, you have to develop that in swift and objective C, so that itself is already different from typical development. And then you'll see automation kind of extend from that. And why it makes it so hard as again, because these platforms are kind of doing their thing, doing what's best for them. But with that comes a slew of tools that are a little bit, sometimes don't quite hit the mark or, you know, what we hear is that device fragmentation is, it can play a very real role in why mobile app automation is so difficult to achieve.

Nice. So I think a lot of times where automation engines get themselves in trouble is they try to test against all the things and all the devices and everything so much.

Yeah, there's so much.

So is there a way to, to know like, all right, here's my risk areas and this is what I should be focusing on, especially with the, with mobile automation, it may be different than browser things I might, I should be concerned about.

Yeah, absolutely. So I think that's going to be more, you know, of your application, how your application works and flows. And so there. We have far different applications out there. You have E commerce applications, you have gaming applications. So playing even just mobile games on your phone. So each application is going to be different in its own way and that's going to need to warrant its own automation to, to work with that. And so that's something else that we're seeing different from web based automation. So if you like web based automation, it sits so easy when, when the web was created, everyone, all the pioneers are working together, they're kind of setting the tone for oh, we'll use this, we'll use this. And everything was kind of all within the same. Right. And so automation following that came easier because you didn't have these say hybrid applications or the different types of applications, you just had one web based application, easy peasy. Now if you were to try to translate onto a mobile application, it gets far more complex with that.

Yeah, that's a good point. A lot of times people complain about browsers saying, well, they don't follow standards. But compared to iOS versus Android, if you're doing scripting is like night and day.

Yep.

So I don't even think it's like that. So I guess that's the first debate when people think about automation is what do they choose? They choose native tooling or do they choose a high level tooling like Appium? What are your thoughts on it?

Right, so it's really going to come down to what works best for, for your needs. And so kind of to, to go back to an question right before. So you're going to understand your needs, you're going to understand your application, you're going to understand some critical components of your application and how that might be working on say an iOS device, on an Android device, things of that nature. And so with knowing that and kind of setting yourself up and understanding the automation that's needed to get done, which would you go for? Are you going to go for native application framework so we have X UI tests that can automate iOS applications or are you going to go another native framework? If you have specifically only Android applications, are you going to go espresso or UI automator? So one thing that I find, and those are going to be faster frameworks because they're kind of working right within the system itself. So an iOS system and an Android system. So they'll be fast, they will be reliable, they might not be. I think one of the biggest things with mobile applications though is that just regardless they can be flaky and really hard to maintain. Flaky being brittle and running an automation and it just failing before it even gets a chance to kick off. So you have These frameworks in place use these native application frameworks that are fantastic again, XY Test and Espresso. So if that, if you have a iOS native only app, then go for X UI test, that'd be perfect for you. If you have an native Android app, then go for Espresso. That'd be a great option for you as well. But that's not so ideal, right? You want to have your application be able to run on all platforms on iOS and Android because that will essentially make your audience far larger. You know, you're not just secluded to one platform or the other. And so that's where we have Selenium coming into play, which I have huge respect for. Selenium, I think they really helped pioneer automation from a web web based perspective. And then they even saw themselves this issue that was happening with device fragmentation. And so they said, you know what, it shouldn't have to be this way. Let's make one more framework. It's going to be a hybrid framework. So what we'll do is we'll make it a simple HTTP server that will kind of envelop these native frameworks. So again, X UI test and UI autom way, boom. Now you have one script to run on both Android and iOS applications. The caveat to that is that when you have that additional step in place by enveloping those, those native frameworks, now we're finding that Appium is far slower than those native frameworks. And in the name of the, you know, name automation, speed is important. It's very important. You want to make sure that your tests are running, you want to make sure, running at a, at a decent speed, making sure that they complete in a timely manner. At Koton, we even have a few customers who refuse to use Appium just because of how slow it is kind of thing. So they're great frameworks out there for everyone to use and it's really going to come down to what works best for you individually. And then there's pros and cons to both of them. Right, so again, device fragmentation native frameworks are so much faster, more reliable, but you're stuck to just one platform. And then here we have Appium combating that device fragmentation, but at the cost of speed essentially. So it's just a little bit slower.

There's a lot there. So I want to pull out a few things.

So sorry.

Yeah, no, that's great. There's a, I just want to pull out, tease out a few things you, you mentioned, I think you mentioned about flaky, flaky automation. I know, I'll be honest, I did automation for over 25 years. I don't think I ever really did much mobile automation. It's all browser based, so I know a browser. The things that make it flaky is usually timing issues on the network or bad locators or timing strategies. It's usually not the tool, it's usually either of those two issues. So what's it with mobile? Is it the same? Is it different? Are there different issues that cause a mobile test to be flaky or less reliable?

More or less the same, you know, depending on your network response. And the tools for mobile applications, they're not as, well, I wouldn't say well developed. They're not as developed as web based automation tooling, if that makes sense. You have far more time between web application development and then the tools that arose for automating that and then the time it takes to really develop those tools specific to any hindrance or any issues going on from web based application. When we look at mobile application automation, those that flakiness that you just mentioned for web based, they still exist for mobile applications. And I think the biggest culprit is, as you mentioned, just the technical identifiers, you know, flaky however you're trying to interact with that element. Even Appium Appium Inspector has a, you know, big old warning sign that says we don't recommend using XPath kind of thing, you know, but XPass is the only thing that you can, that you can use between two platforms unless you try to add your own unique IDs, which a lot of times that will fall on the developer to put those in. But a lot of people don't have that resource to do that and that just puts more time on the developer's hands. So a lot of people are kind of, I wouldn't say at a loss, but are subjected to using XPass. And XPaths in themselves are flaky and brittle and not the most ideal to work with. But sometimes that's the only thing you can kind of work with. And the tooling within mobile application automation hasn't had that same amount of time to really develop compared to web application automation, if that makes sense. So we're kind of still seeing it, the automation that's in its infancy, if you will, as far as what tools and frameworks can allow to kind of combat that flakiness and be able to maintain your scripts.

Absolutely. So I know a lot of companies, or not a lot of companies, a lot of frameworks are coming out now to try to combat a lot of those issues you mentioned. The way they do it is codeless with some sort of AI machine learning behind it that's able to say, okay, this locator is not available, let's use an image or let's use an accessibility identifier. So there's a lot there. But I guess the first part is big buzzword is codeless. So you've tried both either. Advantages disadvantages of a script based automation compared codeless.

Right. So this is where you really see artificial intelligence come into play and it is super exciting. So when you think about, just to get a little background, when you think about the tech trend, say five, seven years ago, what was the big thing? It was the cloud. Everyone wanted to make it to the cloud, right? So now the new tech trend that we're currently seeing ourselves in is commercializing artificial intelligence. Really bringing that into all areas of technology and business and things of that nature. So we're seeing that brought into even the, as you mentioned in the testing and the automation world. So we have codeless, scriptless, no code, however you want to phrase it, automation that is being driven by artificial intelligence, which I think is really, really cool. There are some drawbacks to that. A lot of times some tools might sell you on codeless and I've put it you can't, can't see in a podcast, but putting in quotation marks codeless, no code where it's not true AI. It's more of kind of just stringing, stringing together, what would they call it, drag and drop and kind of stringing test steps together to create a test case. But that in itself can be really, again not true artificial intelligence. And so it can be very inconsistent. But when you do have a very consistent framework in place. So I'll use Cobitant as an example. We do offer scriptless automation and it's true scriptless, it's true artificial intelligence driven just based on machine learning algorithms. I believe we have a TensorFlow algorithm in there that really learns your application and not only learning your application, but learns your movements within that application. And so it's a little bit more robust in that sense of a true scriptless framework. And again, the draw side that I trying to get at, and I keep getting sidetracked is that you are at the expense of the AI driver, right? You don't have any control over that. All you really are doing is providing a manual session. So you're providing a manual test case, if you will. And of course it will save that Store it, learn from it. What is machine learning but just learning off iteration? So every time you use that test case over and over and over again, the AI will actually become smarter and knowing your application. And there's a lot of great aspects to that, but again, you're at the ex. You're at the expense of the AI itself. And scriptless kind of comes from a more top layer down scene, so you don't get as much testing coverage. With scriptless automation, you're really coming from the, from the ui, from the top down. You might be able to kind of dig into some API testing or database testing or end to end testing, but not as, not as comprehensible as if you were to script that out. So, you know, in script based automation you can really dig into it, you can really get to that granular control of what it is you're trying to automate. Script list will save you a ton of time because it's no code. Right. But again, you're at that limitation of coming from the UI down and kind of being at the expense of the artificial intelligence itself. So you'll kind of lose a little bit of that control. So again, some pros and cons there as well as far as when it comes to scriptless automation.

Nice. So I guess for the folks that don't know about Kobiton, right here, Kobiton for some reason I think Catalan Studio, but I don't know why.

Oh, it's our parent company that makes that. Oh, that makes sense.

Okay.

Yeah.

All right. Yay.

Here you go.

It makes sense. That's a rarity. So my question is then how does Kobiton enter the picture? Then what do you all do? Do you add a layer on top of Appium that adds AI, or do you inject AI based on. If someone runs on your infrastructure, how does it all work?

Yeah, so we actually have Nova is our artificial intelligence engine. So we call it Nova. Nova actually sits under, kind of in the between, between the device itself and the application. So it's kind of sitting underneath the application itself and it's really just watching. It's just watching. Capturing a lot of metadata from your manual session and again really being able to learn your application and learn your interactions with application. So that's kind of how our AI engines working at the moment, Kobiton's main tenant really is to go faster, go forward and go flexible. So with our go faster, we actually have our own. So if you want to do scriptless, which is great, we do. Our Nova does capture the XPath associated with every element that you touch. So even though it's a scriptless test case, there is you're able to compile an entire not compile your Nova, it will generate an Appium script for you thereafter. So not only did you just run, you ran a manual session and now you have scriptless and you can kick that off on any one of the 400 devices we have in our public cloud. Again, to really combat that device fragmentation, make sure your application looks great on every platform. And then with our scriptless solution, you can then use that same scriptless test case. It will generate an Appium script, a full Appium script for you of all the interactions that you performed within that manual session. And so boom, you have an Appium script now that you can run again on any device or set it up within your own framework or your own test suites or Jenkins Jobber, you know, CI CD pipeline that you see that you're using and now you're able to leverage your script based automation on top of that. So we really help offer both solutions for our users. We understand that not everyone's going to enjoy scriptless. People are already coming with other Appium scripts and want to be able to use Appium so we can support that as well. So not only are we offering an auto generated script that you can use, we also have our own in house Appium server called Axiom that is actually three times faster than Appium Server. So when I mentioned the caveat earlier just a moment ago with Appian being so great and how it can envelop both.

Native app frameworks, Cobiton can do it in the same, but we can do it three times faster leveraging Appian scripts. So we really wanted to go faster for our users and when it comes into our go forward play. This innovation again on itself is stemming from a really robust AI artificial intelligence engine that's running it, that is constantly being worked on with every release to really make sure that it has the utmost confidence in failures and passes and everything so that we can allow our users to know that they're in good hands using Kobiton.

Nice. That was one of my questions. I know how AI assists with helping tests become more reliable, but I didn't know about the how helps it get faster. And it sounds like you have technology and a grid that speeds it up as well. Did I hear that correctly? Nice.

You did, yes. Yeah, I feel like so you know, say you're setting up your automation environment, you're using Appium and even to start a even to start coding out a test from scratch can take you a good moment. It can take you say anywhere between an hour, two hours of again just laying down that preliminary test script. So using Kobiton and allowing the AI to generate that script for you. So it's kind of like that. I don't want to get confused with any sort of like record and playback features or I know Appium Inspector has record where it will record your movements and it will give you kind of like a little, a little snippet of the Appium code that you can use. Our script is far more comprehensive. It has, it's, it's. Everything's in there that you would need including tear down and scrolling and things that are hard to automate. Like scrolling for example is actually really hard to automate. Um, so with this auto generated script you have all that right then and it takes five minutes to get it together. So you're looking at two hours just setting up a test script versus let's say 10 minutes at most having the entire test script created for you and with a great thing with that. We're not, we're not here to say we're getting rid of, you know, your Appium scripting. We're here to say let us lay down that frame, that foundation for you, you so you can spend more time on the important things like drilling even deeper into that test script if you need to, so to speak. So again it's really laying down that UI level for you, that foundation. Now you have, you just saved yourself an hour and 50 minutes. Now you can use that hour and 50 minutes to really start drilling into that test suite or that test case or move on to the next test case to automate, so to speak. And then if, whenever you're ready to run that, you can run that using Axiom, which again three times faster. So if a usual testing window window takes you six times hours to run all your automation because Appian can be a little slow. Now your testing window is two hours of running your, of your test suite using Kobiton's Appium server.

Great. So I know a lot of people, what they do with AI or machine learning is they also like you said, there's. You have a.

Scriptless approach but also it helps people that actually scripting. So does it have any sort of intelligence while people are a coding that gives them suggestions or a telesense or is it just that it creates the template for them and then they can add on to it based on their needs by coding?

It's more than the latter. So it will lay down like not so much as you're scripting it out, but it will provide you again that foundation to start off of and then proceed with your additional scripting however you see fit. Whatever is needed in that test case, whatever requirements you're trying to make sure you're hitting and passing.

Nice. So if someone has existing Appium test and they wanted to go to say a vendor that has this AI technology like Kobiton, does that work or do they have to start from scratch?

So it comes from both ways. So if you're wanting to do an automated script generator, that would come from a manual session, so that's going to come from a manual session and then it will be converted. We do support bringing your own, you know, coming to the table with your own Appium scripts, but that is not going to work so much. It won't work for the auto generated script, if that makes sense. You can still run it on Kobiton devices and our AI will still take all that, the run itself and all the metadata associated with it. It's just from a perspective of you actually coding it out versus it being auto generated.

Perfect. So I know a lot of things other people struggle with is reporting or knowing when to what test to run based on check ins. Does, does this solution do anything with CICD or to help, you know, like what test to run? If they check in code AB that they need to run test Z and Z and one or something.

Sure. So we do support CI CD tools, so we do integrate with Jenkins and so we are, we do allow our users to set themselves up from a continuous integration or continuous deployment perspective. This is going to start first come for scriptless. So not so much script based at the moment, but we do offer what we call impact or soon to offer with within our roadmap of this year, what we call impact analysis. So whenever you have a new push to test or you know, say some of your save a new push to test that has a new page or a new button or something within the application that is different from the test cases that you. The scriptless test cases that you already currently have.

Nova will let you know the next push a test. Hey, looks like this test case, this test case, this test case has been impacted by this push. Might need some remediation. You may need to add in another test step here just to maintain that that existing test script or scriptless test case to make sure that it's not coherent. But working alongside your applications, you're not missing anything if that Makes sense. Yeah.

Great. Cool. So I'm looking at my notes and for some reason I've written down a two prong approach, something about capture, object inspect. And what is this?

This is the exciting part. This is like. Okay, let's, let's talk about this. Okay, so this is what we are calling it Flex Correct. So Flex Correct is the first of its kind coming to the market and this is where we're seeing artificial intelligence really help and assist script based automation, especially when it comes to flaky tests and inconsistent tests. So I mentioned before that there are just not ideal ways of how you can, how you can implement technical identifiers. You know, like I said, they do have, each platform is going to have their own, you know, say their own find element by or you know, native. Native frameworks will have their own specific locator strategies. But that doesn't, you know, quite help when you have a hybrid application. So you can't use iOS specific locator strategies in automation when you want to automate an Android part of the Android application, so to speak. And when you're using locator strategies like kind of, you know, find element by say class name or id, that also may be platform specific. And so that's why XPaths are kind of a sure way to make sure that it's compatible for both platforms. So you have it for iOS and for, for Android. But XPaths like, like I mentioned before, Appium Inspector even says, hey, we recommend don't not using XPaths, right? They recommend using unique IDs and unique IDs are great. And if you have that capability and that resource to implement unique IDs that's going to definitely make sure that you have the most stable automation. But again a lot of people don't have that resource, they don't have that, that relationship with the developer. The developer doesn't have the capacity to add in these unique IDs. So more times than not you're kind of, let's say stuck so to speak, by using XPass. But you know that XPaths are very flaky and brittle. So Flex Correct, like I mentioned, is first of its kind to the market to assist in script based automation. So that when you are running your Appium script and say it fails because element not found on page Flex correct will actually prevent it from failing and it will find the Next best fit XPath for that element in question. So that, and implement that so that your tests continue running and you're not, you know, your test didn't fail because of the very frustrating and very inconvenient element not found on page. So that's Flex Correct. So that, that's the two prong approach essentially. So that is where artificial intelligence comes into play by analyzing script based automation. So it's capturing the object within your tree hierarchy of your application and then it's also during a manual session or even just reruns of the test case on a real device. Nova Combaton's AI engine can actually map elements across the screen. So if you're tapping this element on the screen, it will pin that to that within the hierarchy and capture that XPath right there. So that's the two prong approach that you were mentioning of how Flex Correct works in the realm of artificial intelligence assisting script based automation. So you have one, one prong that is again capturing the object, so catching the entire hierarchy and learning from it, analyzing it, and then the second prong of being able to map elements, map that element, pin it to the hierarchy. So when you run that test again and say the developer added a new button, added a new dropdown, added something that would kind of knock out any of your X pass or any of your technical identifiers. With Flex Correct, it will correct that instance for you. And so it ensures that your tests continue running, you don't have any failed negatives, so to speak. And it really just again helps facilitate your testing and make sure everything just keeps running and nothing fails due to that pesky element not found on page.

You have the grid, how it makes it three times faster. Do you have any studies on how this type of technology helps your test become X amount reliable or anything like that?

We are working through benchmarks at the moment, but we are looking at more of that confidence interval for the artificial intelligence. How confident are, you know, is it that it can select the correct, the next correct X path? And that is looking at like a 98%, 99% confidence or confidence, not interval, but just confidence in itself of being able to again identify the next best fit technical identifier. Cobiton did do a research study on automation and on, you know, mobile application testing. And what we found was that first of all, just achieving automation from a mobile application perspective is a big feat in itself. So I think only about 20% of tech teams can actually achieve mobile application automation. So let's say, for example, you have your entire regression suite fully automated and you kick it off. Everything passes or not passes, but nothing, you know, nothing fails due to element not found and you can move on next Sprint, the developer, like, pushed something to test, pushed a new button, a new dropdown, and it kind of rearranged that hierarchy. And so now that XPath is no longer true XPath for the original element, we found that anytime a push to test would nearly blow out half of the test in the regression suite because again, just the maintenance and the flakiness is real. We found that it took about, I think, like two weeks to go back into every test script and correct it and to tweak it, maintain, you know, and to keep it, keep it going for the next regression cycle. So which isn't it just. That's insane. I mean, that's like a whole release cycle in itself of having to continuously tweak your scripts and make sure that they're consistently running. So we did do. We do have a research study on that show that it took a good while to correct those scripts. Now, with Flex Correct, you kick off your regression suite. Sure, your aggression cycle time might be longer because as your tests are running and say an element's not found and Flex Correct's doing its thing, it does add to that testing site that automation testing time, because it takes a moment to do its thing and again, find the next best fit technical identifier. But at least you don't have to completely fail your tests again. It'll keep your test running and it. That maintenance window goes from two weeks to maybe two days. And it's just, it's, it's, it's wild. We're really excited for this one.

That's awesome. I know a lot of people that's. Alan Page is telling me something called Meantime to Diagnosis. Is that how long does it take you to debug your test?

Right?

If it takes you, like, hours, then you've lost, right? Because your next build's already kicked off and running, you know, so it's like a maintenance nightmare just to keep up. So this, this is cool. Very cool.

Absolutely. Yeah.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode, head on over to test guild.com a385. And while you're there, make sure to click on the Try for free today link under the exclusive sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show, and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe, My mission is to help you succeed with creating end to end full Stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation podcast. Head on over to test guild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.