AI Augmented Testing with Avinash Tiwari

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey it's Joe and welcome to another episode of the Test Guild Automation Podcast. Besides the podcasts and online conferences that I host and run, do you know we also have pretty much every week free training session webinars that you can attend from the Test Guild. If not, check out testguild.comwebinars to see past and upcoming free training. So today I want to share with you a very popular webinar we recently had with Avanesh on AI augmented testing for digital apps. If you don't know, Avinash is a renowned thought leader, sought after keynote speaker and co founder of pcloudy.com he has over 20 years of experience in product development and testing and he brings with him a passion for emerging technologies and quick adoption, both of which have really solidified his reputation as a leader in the mobile app testing field. And as you probably know, with the ever increasing need for high quality releases and feature rich digital apps, the demand for faster app development life cycles has become non negotiable. Today the world is racing against time to get the best out of their apps and this really has become a competitive edge for a lot of companies in the app market. While implementation of various testing methodologies like DevOps, continuous integration and delivery have helped accelerate the process, there's still a lot of scope for improvement. Of late, artificial intelligence based augmented testing has taken center stage to show us ways to enhance and accelerate our efforts. So in today's episode, Avanesh is going over three main ways. AI or artificial assistant or augmented testing could start helping you with your testing efforts right now. And if you don't know, according to Gartner, the use of AI can dramatically increase the efficiency rate of test automation tools. And this gives us all the more reason to quickly adopt new methods and ways to explore AI augmented testing to see how it can help us scale our testing efforts to to build and release higher quality apps faster. If you're listening to the audio version of this and you want to actually see all this in the video version, all you need to do is head over to test guild.com forward/a4:40 and you can check it out in the show notes.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with cloud confidence at every step from code to deployment. For Every framework, browser, OS, mobile device and API get a free trial. Visit testguild.com sourcelabs and click on the Exclusive Sponsors section to try it for free today. Check it out.

Hey Avanesh, welcome to the Guild.

Thank you so much Joe. Great work by you and your team in creating this community where we can share and learn and share ideas. A very warm welcome to everyone who is part of this session. I am so glad to see so much interest in this topic. The last few years we have seen two Black Swan events. The first one was Pandemic which really pushed the digital acceleration across enterprises across across all kind of businesses and this acceleration happened at a breakneck pace which was never seen before. The second Black Swan event happened few months back with Open API launching ChatGPT and this is turning out to be a huge step in democratizing access of AI. So this talk today's talk is kind of an intersection.

Of both the events in the context of app testing. So we're going to talk about AI augmented testing for digital apps. So over next 30 to 35 minutes I have plans to cover following. The first is how AI is changing the tech landscape at a very high level.

What are the testing use cases that we have seen in the industry our enterprises or our customers are using right now? And towards the end I wanted to focus more on how we are leveraging AI in P Cloudy to solve some of the testing challenges for our customers. So with that first thing first I just wanted to have a quick intro for anyone who is not familiar with P Cloudy. PCloudy is one of the leading platform for mobile application and.

Web browser testing.

Started in 2016. We have grown very fast serving more than thousand plus customers across the globe. Our customers have run million plus automated tests in real environment.

Consists which consists of close to 2,000 plus devices and browsers and our ratings that you see on the slide is reflection of amazing love that customers have shown towards us. So we are one of the highest rated platforms on platforms like G2 Crowd and Gartner Peer Insight and if you have not tried pcloud yet, I would urge all of you to go and take advantage of the free signup that we have. Go ahead and try and share your feedbacks and we'll be happy to help you there. With the introduction out of the way, let's delve into the actual agenda of this session today. This is me in 2018 when I was giving a talk in one of the largest testing conferences in India and the talk was around how to reimagine the future of testing through AI and it generated huge buzz in the community. Post the session, a lot of people came to the booth and interacted and asked for suggestions on how to implement where to implement AI within their testing cycle. But the matter of fact was that AI at that point of time was quite difficult to implement. The models were very complex to build, data collection was a huge challenge.

And the success of AI projects were in doubt as well.

Fast forward to 2023.

Things have changed dramatically and as the cartoon rightly predicts, you can go and ask AI to give you precise answers on where to use, how to use and then use the same models to build AI for you.

So with chat, GPT and open APIs, pre built models and there are so many other such platforms now available for us to use this, pre built models are giving us access to the complex algorithms behind the scene on a single click and using that.

Implementing AI I feel was never easier than this. So we have already seen so many chatbots coming into play across different use cases, Auto generation of code in context of testing, manual.

Test case generation, etc have become so much easier than what it was maybe a couple of years back. So we are truly living in great times where we can leverage AI for our testing space as well. And this is impacting everywhere, right? I mean some of the data points that we have seen from the reports also suggest that so IDC reports suggest that 75% of the large enterprises will rely on some form of AI processes to improve their efficiencies, to improve product quality in all possible ways. And same data point reflects in Gartner report as well. When you ask this question, which technologies are most likely to be implemented within your organization? AI ML is one of the primary technologies. So look at how the industry is shaping up and, and some of the reports reflect that trend as well. Now we did our own research in last six months and we published a report on AI driven testing trends for 2023. It's a very detailed, extensively done paper or a.

Or to very frank a white paper where you can go in detail as to what are the different use cases some of the industry experts are using or suggesting for for 2023. So I would really urge all of you to go and download the report. It's available on our website pclary.com and take advantage of that. But there are some snapshots from that report I would like to highlight here. One of the key insights that we got in terms of which are the areas where AI is being used. So of course visual AI is on top of the chart visually. I has been there.

In this testing space from last few years and a lot of evolution has happened. There are much better API is easily available for you to implement visual AI testing within your.

Test cycles. So definitely visually comes top of the chart which is no surprise. But then there are a lot of areas where the focus is on improving the automation script creation and automation maintenance. So creation of automation script object identification, self healing, etc. Are the areas focused in the test automation space where we see a lot of AI use cases being implemented and then there are quite some interesting ones where it's about configuration, predictive analytics, impact analysis where customers have already started using it. So diverse set of use cases where you can think of within your context, within your organization to start using it. But again, if you want to look at the detail of these use cases, you can go ahead and look at the report to be more specific on.

The use cases which are experts suggested. These are some of the use cases from the report listed without any order here. So very interestingly things like synthetic test generation can be done very effectively through UI visual testing. We talked about bug classification, a big area where AI can help server configuration planning, auto trigger of test auto complete auto code completion. There are so many areas where customers have already started using and if you have not thought about it again, please go and look at the report and you'll get some deeper insight on some of these cases.

So with that what I wanted to talk here is more on how we are leveraging AI within P Cloudy. These are some of the areas where we have already solutions available. There are certain newer areas that we are working on especially with all the generative AI becoming.

Easier to access. Things like code completion will be coming up very soon. But there are three areas where we are specifically very strong at first is increasing your test coverage. Most of the automation today is in form of either functional or API or unit tests. But how do you add a complete visual validation on top of your automation test cases where visually comes into play. So it's a way to enhance your test coverage without kind of changing too much within your existing scripts. So that's the first.

Section or first solution we want to talk. I want to talk about. Second is the self healing automation scripting execution we all know has been going through this pain point of failing often. How do we maintain it? So cell feeling comes as a as a solution where at least part of your maintenance can be done in automated way and as it is evolving, majority of the automation.

Maintenance can happen through self healing, but it has still some way to go. So self healing plugin that we have released can really help you improve your automation speed. And third one is a test bot which is a completely autonomous bot which is responsible for speeding up your test cycle. So you can use it to do a lot of random tests on variety of devices and browsers and take advantage of the report that generates. It generates very detailed reports. So we'll go through these use cases bit in detail. But as I said three areas increasing test coverage, increasing automation speed and speeding up the test cycle is something we are focusing on right now. So let me switch on to a completely different environment. So what I want to show here right now is one of the case about a cell filling example. So here is an application which is a sample application created by us. A simple page which is a terms and condition page which with two buttons accept and refuse. And what I want to highlight here is let's say we are planning to perform an action on this accept button and this has certain properties based on which the action will be performed through Appium script. So you can see it has a resource ID and with the text except now what's what typically happens we have seen so so many such automation projects is that when the next build of the application comes the property of the the locator changes and your script fails. So that's exactly what happens here as well. But before that, here is a Appium script. So on the top what you see here is P Cloudy capabilities. So it means it's designed to run on P Cloudy platform and but that integration is just adding the capabilities. Remove that you can run it on local environment as well. And then a simple test case where we are clicking on this accept button.

Nothing fancy here, but let's say.

We plan to run this test as a Junit test but you can use any kind of framework. This is not limited to Appium Junit framework. You can integrate this plugin on any other tool as well. But right now it's for Appium and Selenium. So here what happens is when we are running there is a new build available and this build resource ID has changed. So you can see here now the the resource ID has ACC instead of accept and the changes could be in other formats as well. For example, this locator itself doesn't exist. So all that kind of object identification challenges we have seen especially during execution. The question is can this script run on its own? We are not making Any change to the script, can the script run on on its own? So we are running the script here and on the right side you'll see the log and on the left side what you see here is a pCloud interface. So here we are also from the pcloud interface you can just open the device and start seeing the live execution. Since it's a very small script, the execution will finish very quickly. But as you can see here, it's running on a Samsung device. So the page is open and on the right side you can see the logs.

So here the page opens and it clicks on the Accept button. So it didn't fail the script. It was able to click on the Accept button and on the log you can clearly see that in the first iteration it failed. It said, it says that an element could not be located on the page using the given search parameter. But then it tries again and it does the self healing. It changes the locator on its own and is able to perform the action on the application. So this is one example, a very simple example of a self healing where you can go ahead and.

Run your scripts without worrying about the changes in the application. And this happens through the. The there is a process that is in place where you have to run the script few times to make sure that the all the locators are in place within a database. And then the next time when it runs it is able to figure out if there is any change in the locator. It is, it can get, it can heal the script itself. So there is a mechanism that that is in place. There's open source project Helenium that we have taken and modified for our purposes. But you can also for you it's completely abstracted. You can directly use our interface and need not worry about this setup at all that we have done it for you within the pCloudy interface itself. So you can just run your script. You will have to just run few times to train the script. But after that it can do the self healing on its own. So that was the first example. The next example that I wanted to show you was around Visual AI. So again I'll take the same example of.

Terms and conditions.

Application. So here is another Appium script very similar to what we saw. Again P. Cloudy capabilities are in place. And then what we have done is for the same application that we just talked about. We have taken the base image of this page which has let's say some text terms and condition, the detailed text and then accept and refuse button.

And what's going to happen is in the next build some of the text of this application changes and also the color of the refuse button changes. And we'll see whether we can identify the differences and how good it is to identify the visual differences when when the script is run on a new build or on different set of devices where we have seen the alignment is a problem or some text gets hidden. All those issues crop up during the cross compatibility testing across devices. So the base image is already uploaded in the script and here is.

On the bottom you see a test function and within the test function there are certain commands that we have given which are pre built command. You can just insert it within your script without impacting any of your existing test cases. So it can be done at your page object level or it can be done at your specific objects validation step. You can choose to do that, but these commands are available very simple to just integrate within your appium scripts and you're done. Once that is in place, you can just run the script and we'll quickly see the results. So same process, you can see the live view happening in pcloud interface while you run the script and it can happen in parallel devices together. So right now I'm just showing it on one device. So again just a simple script where it opens the page and then validates the ui. So when the script is completed you can come back and see the result in the in your framework. Again there are various kind of reports available, but this is a simple example of you see the report within your framework itself. So here you can see in this new build the condition text is missing, it's only terms. And while in the original build it was terms and conditions. And then the refuse button has a different color. While in the original build it was gray, in the new build it is blue. So this is the result where you see the difference between the base image and the image which was taken at the runtime from the device itself.

And on the build which was used for the execution. So it's clearly marks that this is the place where there is a difference and this is the place where the button has a difference. Now right now it's just mentioning that there is a difference. It's not telling you whether there is a color difference and things like that, but you can enhance the script to.

Be where the script can start suggesting you what are the differences itself. So it can be enhanced, but in a simple way you can find the exact differences.

At the UI level through the visual AI plugin. And the best part is you don't have to spend, I mean on, on licensing of this plugin itself. It comes as part of the Peak Loudy package itself. So very simple to use and, and without any additional cost. So these two.

Combined together Self Healing and visual AI can really improve the test coverage and also improve the speed. So with that I'll go back to the next part of it, the third use case that I was talking about. So third one is more of a automation bot where you can just pass on your application, you can upload your application and provide certain parameters like login information or what kind of device metrics that you, that you want to use. And once you provide that, the bot does the testing on its own. You don't have to write any script, you don't have to provide any other input. So it tries to navigate like end user does on an application. And while it does the navigation, it captures various parameters. So for example, you can see this test was run on 10 different devices across different versions. And then it captures different parameters like how was the CPU performance, memory, battery and it also captures the functional flows it goes through and screenshots of all the steps that it executes. So it's very good to use this bot when you are kind of planning to push your application to a final release on, on on let's say App Store or Play Store to do a final check across variety of devices and see if there are some crashes, some random failures. A lot of customers are also using it to do sanity tests before they start a more detailed regression test on their applications. So various use cases you can think of utilizing this for. So three different areas that we talked about. One is visual AI, second is self healing and third is autonomous test where P Cloudy has been able to successfully use AI for your augmented testing. The whole idea is that you should be able to do more than what you are doing through some of these additional plugins or bots that we have created. And as I said, we are coming up with more such additions to the product where you can do more AI augmented testing through through the use of AI based plugins or bots. So in conclusion, just to end this session, what we talked about.

It'S clearly visible that AI augmented testing can enhance your test coverage, efficiency and speed. It's no more just in theory.

These solutions really work and we have seen huge advantages. Our customers have been able to take from some of these plugins that we have created. And a lot of cases customers have built something on their own because now the models are so easily available. The image difference APIs are so easily available, so you can think of building on your own as well. But if you don't want to do that, you can take advantage of something.

That P. Cloudy has created. So if you have not thought about where to use, I think this is the right time to start experimenting because there hasn't been a better time to include AI within your testing vision or testing strategy. And if you want to know more about the AI use cases, I would really urge all of you to go and download the report from PCLaroo.com you'll find it on the homepage itself. So with that I will conclude my talk and session. Thank you so much for listening. Hope to hear more from you. As I said, there is a free trial available. You can go and sign up. We'll be more than happy to help you. I'm really keen to take up any questions that you have. Thank you so much for your patience.

Thank you Avinash for your automation awesomeness. For links of everything of value we covered in this episode, head on over to test guild.com a the letter A followed by 440. And while you there, make sure to check out our awesome sponsor Sauce Labs by clicking on their Try it for Free Today link and learn more about their fantastic products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed in creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@test guild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.

Sam.