AI based Autonomous Testing Lessons Learned with Shani Shoham

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Shani all about AI based automation testing. Shani is the CEO of 21 and is a serial entrepreneur currently on his third team testing company. Over the years, he's worked with engineering and QA leaders at organizations such as Ford, Walmart, Verizon, Salesforce and many others to optimize the development and quality efforts. He has seen the industry evolved over close to a decade and he really has a pulse now on this type of AI autonomous testing that more companies are coming up with. So really excited to talk about him, to talk with him more about this topic. You don't want to miss it. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Shawny, welcome to the show.

Hey Joe, thanks for having me. How are you doing?

Great, Great. Great to have you. So I know AI autonomous testing is really a hot topic, but before we get into it, is there anything in your bio that I missed that you'd like to add to the guild and.

Share with the Guild based on the accent, you guys will probably figure out in a minute or two. I'm waiting. From Israel? Yeah, that's about it. Been here in the Bay Area for the last 10, 11 years. That's about it.

Awesome. A lot of new innovation I've been seeing, I've seen for a while has been coming out of Israel. It's almost like the Silicon Valley and the Middle East. So you've been involved with three different companies. Why testing and why do you think that area brings up so much in this space?

So I think, you know, kind of as I look at how software has been developed in the last, I don't know, 20, 30 years. And I've been developing software and writing code in my, you know, even going back to my days in high school with Logo and Visual Basic and then COBOL and stuff like that. So I think there's been a lot of innovation in the areas of writing code, in the areas of like repositories, collaboration and really I think kind of the main bottleneck for organization today is really testing. And so I think finally we're starting to see some light being shined on DevOps and testing.

So now at a high level, how would you explain AI based automation testing to say, someone that's a skeptical tester?

Okay, so over the last couple of years, my kind of my opinion and my definition of AI based testing has really changed. And I would say first of all, the problem that we're trying to solve is the fact that tons of hours are invested in every sprint on qa. Whether you're doing that, whether your QA is automated, or whether your QA four or five years ago I was a big proponent of autonomous testing, which is essentially kind of testing fun, very little human interaction. And with human guidance. I've seen a lot of test scenarios to come to the realization that just having a machine spin up a bunch of tests is not really realistic. And I look at kind of the scenarios that our customers are testing today. So there's testing stuff like negative scenarios, they're testing stuff like permission based scenarios so one user can do certain action. Another note, testing stuff like you added a certain field and you want to make sure that it shows somewhere in the application. So a lot of things that more than just plain vanilla flow, which is probably something that autonomous testing can probably adapt to. And so I've come to the realization, I mean my definition of AI based testing is kind of a guided authoring that's driven by AI. So AI actually helps you write the test. And in a way that that of writing that from scratch, it gives you a baseline which is kind of what 21 does instead of, you know, as you kind of add actions, you try to suggest additional actions that usually come along. And essentially it tries to be kind of one step ahead and help you write your test. But at the end of the day, you kind of define the scenario. And if you think about it, at the end of the day QA is about there's a certain initial state, you're performing a certain affections and you're expecting a certain result. And I just, I think that the QA person usually has definition of what the outcome should be or what the end result should be. And by the way, you can take the same scenario and five different QA people and they would each assert for something very different. So I think there's always going to be some kind of human QA person involvement. But I think you can take an automated test that takes two or three days to write. You can narrow it down to a few Minutes to maybe an hour with something. That's where machine essentially helps you fill in the gaps and make sure that the test that you're writing is stable, etc. So that's kind of my definition of AI based testing. But at the same time, I think there's a huge opportunity in eliminating a lot of the QA time that we spend today using AI.

Absolutely. And I also believe you speak with a lot of customers, a lot of other testers. I know you demoed an earlier version of something you were working on maybe six months ago. I think you talked to a bunch of people, you show me again six months after, and it sounded like you pivot. You did a pivot. So what was something that customers, customers told you that you thought they believed about AI, and then after talking with them you're like, oh, actually that's not what they want. This other thing.

Yeah, great question. So that's kind of in line with what I mentioned earlier. So first version was essentially a version of autonomous testing. Spoke to a lot of relatively large size enterprises. And what they said, listen, I this autonomous testing, I mean, it looks great, but I don't know what it's really testing for. And at the end of the day, they each had an Excel file, a document, some cases, some kind of test management system, and they had a bunch of scenarios and said, okay, these are the scenarios that I want to test for. And so that kind of led us to pivoting to, I mean, we figured out that based on the autonomous testing, gave them kind of a graph of the application. So in a nutshell, we modeled the application into something very similar to a sitemap. So we know which screens connect to which and what's the navigational item that connects between them and what's their functionality on each and every screen and what that allows us to do. Now, I mean, you write the test and you have like an editor or a studio like experience. But at the same time, if you just tell me, hey, you know, I want the flow that connects, goes through the loading screen and connects the home screen and then edit profile, and I already populate the baseline of an action, you can still customize that, you can change the data, you can add assertions, but I give you a baseline of a test that's, you know, pretty much 90% done. And so kind of my realization at the end of the day is that every organization have their own scenario they want to test for before they, they release something. And so we had them write own scenarios, but we do that in a way that's very guided. And again, kind of gives them a baseline with just a few data points and so. And then we enhance that to also help them maintain those tests and kind of identify how changes in one test might apply to other tests and autonomously add those actions and a bunch of other things. But that's kind of the main pivot, I would say. It's just shifting from autonomous testing where there's a black box, does something and you don't really know what's being tested, to point where you have a studio and an editor that you give it a few data points, it spins off a test or a scenario that you can start customizing. And under the hood we kind of take care of all the locators and other stuff that make this stable. But the fact that we analyze the application ahead of time makes it easier for us to kind of give them that baseline and take off all the locators underneath.

So there's a lot there. I like to dive into the process. But initially how does that work? Do you have some sort of intelligent crawler that crawls the app? Then based on that, it creates a baseline. Then people can then go in and say, here's what my test would be based on these assets the Corolla picked up. Yeah.

So at the end of the day, the kind of to auto test, you have a bunch of screens that you choose in order to define your scenario. We have two type of customers. We have customers that have autonomous license enabled, in which case we have actually a caller that calls for the application. But as opposed to kind of testing the application, it tries together the, I would say probably the fourth or the fifth level of screens. And while it does that, it captures the object, we captures the active element, captures the navigation between one screen and another. And then we have customers that don't have autonomous license. And what they do is essentially kind of our assumption is that you don't have a mobile device because most people only have one device. And if they want to test an Android and they only have iOS then other kind of stuff. And so essentially we give, we streamline the upload application, we streamline the application in real time and we let them white test kind of while interacting with the application. While they do that, they actually teach our model and they will start building that graph. So there's two models to get the screens and the edges into the system. And once you have a certain amount of test or a certain amount of screens, then most of the point from that point is you have the screens, you Just flip through the various screens and you kind of choose your flow on the go.

Nice. So you said it does generate a lot of these screens and you did mention if something changes, it makes it easier to update. How does that work? If. Does it have a screen like in a map, and then if that screen's touched, say by a developer, and that screen shows up in 100 tests that it would flag to say, hey, this has been changed. How does that work?

I guess the question there's two type of changes. So there's changes to you make your minor change to an element or you change login to sign in, for example, change locator. Honestly, some applications don't even have locators. And it's something we can talk about later down the line. But a lot of the frameworks that we actually, or all of the applications that we're testing actually have frameworks literally with no IDs. But for those type of changes, we have seven, eight different safeguards. We look at, we have an algorithmic system that weighted scoring that looks into stuff like accessibility ID and of course resource id, but content description, class, the value of the element, the position on the screen, the bounds, and of course, you know, you need to convert that from one screen resolution to another. We also do some use some visual analysis. So any change that you make to an element, we're most likely still going to be able to find the element based on one of the attributes that I mentioned. Another type of change is a change to the flow. And so today, if a flow changes, you only need to change one test. And if you think about the way we kind of model the application, we model that into screens and edges. And so if I know that I have a login screen connected to a home screen, and now we change one test and you added some kind of tutorial, some kind of, I don't know, some kind of screen in the middle, then I can identify other tests that have the same pattern of login screen connected to home screen. And I can suggest to actually add the missing screen, including all the functionality in between. And you can decide if you want to apply those changes kind of autonomously or you want to apply those changes as tentative, which is something like bug changes. I mean, we'll keep those changes in the test, but we won't apply them until the user actually confirms those changes. But I can tell you literally yesterday we got an app from a customer. They rewrote the entire app. And I would say we had about 20 tests or so. Took me about an hour and a half to kind of change all the tests, those, those changes. So they change the flows, they change the screens, they change the locators and a lot of that we could handle ourselves autonomously. And wherever I needed to change something, I just changed the flow in one test and then a bunch of other tests that got autonomously updated. So that's pretty cool.

That is cool. And you did mention about like what happens when there are no locators. So you have AI technology then that is able to interact with elements dynamically if something changes. And it knows that this is now the name field, even though the locator is no longer there or something like that.

Yeah. So we have a bunch of applications where it's essentially view groups and it's cross platform applications. And so there's no resource IDs, there's no content description. The way we, as I mentioned, there are a couple of safeguards that we use, including looking at the bounds of the element and the position on the screen, as well as visual analysis. And when you think about it, kind of those three methods are independent of any location that you might have in an application. So that's how we use, that's how we test applications that are cross platform and don't have any unique IDs. And it's working pretty nicely. One of the challenges we faced early on is kind of how do you convert that from one screen resolution to the other. And so that's where we had to build an algorithmic mechanism to be able to convert those bounds and those positions on the screen from Pixel to a Pixel 3 to Pixel 3 Excel. And that took a little bit of time. It is working very nicely and it actually executes pretty fast as well. So one of the challenges with those applications is even if you have IDs, they're just like pulling the object through. It just takes a long time. And getting API responses just takes a long time. And so that's another thing that we've been able to optimize along the way.

So another thing I think you talk about is results you can trust. I think one knock against AI based solutions is testers say or management says, how can we trust the results? So what do you feel should be baked into a solution like this that gives you confidence in knowing that it is doing what it says it's doing.

So first of all, I don't know. The data point for me is say at this point we probably discovered hundreds of bugs. And usually when we kind of interact with customers for the first and second time, the first scenario that they want to see is actually a scenario. They know that they have a bug in the system and they want us to reproduce that bug. And I can't think of a single case that we weren't able to do that. Again, it goes back to my definition of AI. When it a black box and it's autonomous testing, then it's very difficult for you to trust the results. But when you're plugging in the assertions and you're defining what success looks like, and the AI essentially helps you write the functionality, then you can really trust the results. But also on top of that, we just, when a test fails, we bring a lot of rich information like log cats and appium logs and videos and screenshots. And so there's actually a lot to work with to, you know, to get the level of comfort that the test, test failure is actually a bug. And I would say probably after the first couple of times they get the conviction that there's actually a bug and then it kind of, that concern shies away. But I think at the core of it is we're not a black box. You have full flexibility of defining the data, the flows, the elements you interact with, the assertions that you assert for. And so it's really up to you to define what success means. We just help you write the test in a few minutes as opposed to a few hours or a few days.

Very cool. And I think, and I talked to a lot of people, so I hope I have this right. I think it was you. Did you say you actually are able to upload some apps and test them before you go to a customer and then find actual bugs and that helps build up trust as well?

Yeah, actually I did that about three times this week. So first of all, the fact that I can, I mean at this point I would say I can probably write the test in about eight minutes. And so I literally had a case last week where in this case that they remote the application. It was still not in the app store. So they had me send the application. They sent me a flow and it literally me 8 minutes. In 8 minutes I replied to the email and I said done and I sent him a video. But you're right, in most cases, especially for Android, I mean the APKs are readily available at the end of the day, you know, we test the rendered version of the application. So we don't have, we don't touch the source code or anything. So I can just take a package application. I usually google the app's name and I download the apk. And then again, it literally takes me a few minutes to write a scenario. And so whether it's an onboarding scenario or any other scenario. Yeah. And we just, I just create a test, I run it, I send them the results in a nice looking video. Usually I look at the reviews on the app store as well and I kind of, I kind of see what users are complaining about so I can actually reproduce those scenarios. And yeah, you know, kind of between the video and a screenshot of one of the recent reviews, that kind of, that's enough to get them, you know, convinced to at least look at what we do. So yeah, that's. I'm kind of revealing my secrets here.

I think. Also when we were speaking before, you said one of the biggest point you heard from customers was automation was test maintenance. And it sounds like you built that into the solution. Another thing I hear people struggle with is test data. Do you do anything with test data using AI to help folks test their applications better?

Yeah. So one of the things we can do is we can autonomously generate data. We build a capability to essentially look at the elements. So if you have like a text, nadi text or something like that, we just look at keywords, either that specific locators or neighboring locators and we figure out what the context of the input box is and then we can essentially generate data based on that. So we know how to generate stuff like first name, last name, email addresses, zip code, city, phone numbers, bunch of other types of data. And of course you can create that in a certain format. So if you want to append a randomly generated first name and last name to a certain specific domain name, so you can identify that in your system and you can do that as well. You can also write custom code. So it's pretty robust in terms of the way that you can manage data.

Now what's used under the covers, does it use Appium or Selenium or is this like totally proprietary? When you write a test, you can't really modify it. How does that work, test creation wise?

So under the hood we use standard Appium, which actually means that I can integrate into any Appium grid. So I mean we have customers running on Perfecto, Source Lab, Cobiton and almost any Genymotion, almost any grid provider. And it also means that it's relatively agnostic to whether you're running on an emulator or a real device. We had to. Real devices actually have some other challenges that you need to be aware of. But under the hood we want standard Appium. The challenge is we do a lot of challenges with the way that Appium works. There's a lot of edge cases with Appium. Stuff like doesn't give you the, it doesn't return the element or it doesn't give you the object tree. API calls with Appium takes sometimes minutes. And so we have to face a lot of these challenges and overcome them to actually optimize the Appium execution. Another thing that's worth mentioning is these are not static scripts. So essentially every time we execute an action, we actually do an analysis of the DOM because we have to make sure the next screen was actually loaded, didn't load, and we make sure we synchronize between the actions and the screen loading time. And then there's the locator. So again, we were able to test applications where the locators are constantly changing, but that means that you can't really execute a static version of the screen. So we actually create those scripts on the fly and while we do that, we analyze stuff like system messages and we're able to handle system messages to make sure that your test doesn't get interrupted because there's a Google account on the device or there's a new OS upgrade that just popped up in the middle of your running test. But that's one of the reasons why those scripts are not static and they actually created on the fly.

I actually realized we've been talking for over 20 minutes here, almost 30. I never mentioned the name of the solution. Is it call 2121. 21 Labs.

It's called 21. When I get on the call, I actually say, hey, my name is shania and I'm 21 and nobody really believes me and you can't really see my face. But yes, based on the voice, if you understand, I'm not 21. But yeah, the domain is 21labsio. And yeah, that's a good point. By the way, for your subscribers, we're in private beta, but if someone is going to either sign up to the website or if someone's going to send me an email, my email is Shania21LabsIO and mentioned that you kind of came to 21 through Joe. Then I'm personally going to follow up with you and make sure you get access to all part of the beta.

Oh, that's awesome. That was my next question. How long has 21 been out and can people get a hold of it? So it sounds like it's in beta, but they go to 21 Labs IO, they could sign up for A private beta. So Shani, what am I missing? Is there any other thing that's packed in here that you think is unique to your particular solution? AI wise, I think the only other.

Thing that I'll mention is, and I kind of forgot alluded to that, but there's a lot of testing solution out there that requires you to install mobile client or an SDK to get an invite from someone to sign up with your EMIE and those sort of things. We actually assume that you don't have a mobile device. And if you think about it, which device, Joe, do you use?

That's an iPhone.

That's an iPhone. And so if tomorrow you have an Android application, you wouldn't be able to test. So we assume that you don't have a device. And so the experience with us is it's fully sas, it's all cloud, you just sign up, you upload your APK or dot the app and that's it, you know, from there you just start writing tests. We already print together with a bunch of devices and emulators. We give you access to those so you can literally, you stream the device, you write the test, you run the test. It literally takes a few minutes between when you sign up and you actually have your first test executed. You don't have to worry about where I'm going to execute those tests and spending months of integrating to one device lab or another. By the way, if you can integrate 21 into any other cloud based lab, as I mentioned, we have customers using Effecto and Source Labs and Kobiton and many, many others, but we've already taken care of that integration. So the only thing that you'll need to do is you'll need to give your credentials and maybe the endpoint and the token and that's about it. And since different providers actually have different API calls, we took care of all of that under the hood. So the first thing that I'll mention is kind of from the point that you sign up to the point that you have a test, it's a few minutes and you get up access to a lot of pre integrated devices. The second thing that I'll mention is when we started 21, we had a bunch of customers that actually wanted to run on real devices and I alluded to that earlier today. Running on real devices is a whole different ball game than emulator and emulator. You always get that at the same state and there's no disruption, et cetera. A real device, maybe the device wasn't clean, maybe there's another user on the device again, suddenly you get all kinds of system pop ups, et cetera. And so, so we build an inventory of system messages and every step that fails we actually run root cause analysis. And if we detect that the reason that it failed has to do with the system message, then we know how to handle a lot of these system messages. So that ensures that if you're doing a nightly regression and suddenly Apple decides to announce a new OS or anything like that, then tweet wouldn't fail. We'll actually, we'll know how to enter a lot of these edge cases. That's just part of wanting real devices to emulate. There's a lot of other stuff, but I think given the time that we have probably covered at least the gist of what 21 does.

Perfect. Okay Shani, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts? What's the best way to find or contact you?

So my email address is shani21labsio Find me on LinkedIn, Facebook, Twitter, Merritt so don't look me up on Tinder. One piece of advice. Well actually maybe that's talk to a little bit about kind of the way that we see things down the line. Our bigger vision is to connect testing and production. And so I think there's a lot of information in production in terms of how they're using your application. You can look at that data and one, be able to correlate between what you're testing and what's in production and see what's your real coverage. And two, be able to figure out what's the high priority scenarios to automate. And high priority is one key functionality, but two kind of the most common flows in the application. And so unfortunately right now this information is with product and marketing tools like Mixpanel. Our bigger vision is to actually bring a similar experience to testers and engineers. And actually we're pretty far along in terms of being able to create that correlation between testing and production. But I think the bottom line is there's a lot of data in production that you can use to enrich your test. So yeah, so that would be my one piece of advice. Advice is make sure that what you're doing correlates to what's happening in production.

Thank you Shani for your automation awesomeness. Links to everything value we covered in this episode. Head on over to test guild.com a308 and while you're there, make sure to click on the try for free today link under the exclusive sponsor section to learn all about Source Lab's awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.