AI Impact in Software Testing with Gaurav Mittal

In a land of testers far and widely journey seeking answers, seeking skills, seeking a better way o the test Illinovation Testing Podcast Guiding testers with our nation awesomeness. From ancient realms to modern days, they lead the way. Oh, the testicle of nature Testing Podcast.

Hey, how will AI impact your career in software testing? Well, you're in for a treat if you don't know because today we'll be talking with Guav all about it all about AI software testing Shift left all the things really excited to have him on the show. He's a seasoned data science manager at a large, large enterprise. He has over 18 years of experience and he's really helped a lot of teams develop and deploy cutting edge technical solutions. He's written a bunch of different blog posts that we're going to highlight here as well. So if you want to learn about the impact of AI and software testing, you don't want to miss this episode. Check it out. Visual bugs are silent revenue killers. Do you know that 88% of users won't return after a poor experience and nearly 40% abandon a site if layouts or images fail? As a tester, you know functional testing alone can't catch these issues. Especially with today's modern complex browsers and device matrix. That's where BrowserStack's visual testing suite comes in. Trusted by Canva Intercom and more, it ensures pixel perfect UIs with AI powered visual testing across 20,000 real devices and browsers from app Percy, from mobile apps to automated cross browser testing, it's a one stop solution for flawless UI designs. Integrate easily with GitHub, Jenkins and more. No installation needed. Start shipping perfect UIs today and see the difference for yourself by heading over to Test Guild Me Vista or use that special link down below. Hey Rav, welcome to the Guild.

Hi Joe, thanks for having me. I am a big fan of you and the podcast. Like they are very knowledgeable.

I love that. Thank you so much. I appreciate it. So I love the knowledge you've been dropping lately as well and I'd like to dive into it I guess. Before we get into that though is always curious to know how did you get into automation testing?

Oh automation testing. Like right from the very beginning of my career I was being put into testing. I love like finding the issues and making sure what clients we are passing that should be of robust quality. Now eventually what happened like this manual testing was taking a lot of time and automation testing it started creating the waves. So it was like white box testing where you were supposed to write some Piece of code to make sure your manual testing efforts they can be reduced. And that's how I got fascinated about it. I love coding. I started my career with Java and written like code like Selenium was there. So I have worked a lot on Selenium Cucumber and Recently from past 3 to 5 years it's like AI wave. So how AI can be utilized into the automation testing. And I have written several automation utilities provided some innovative solutions to the organizations which were very successful. So this has been my journey from manual QA to automation and now leveraging AI in the automation and some approaches.

Very nice. So what are some of those automation utilities?

So automation utilities like couple of those I would like to talk like one of them is like how you can categorize the emails for example. Like most of the organizations they are service based organizations and they provide their services to product based companies. These product based companies they then have a lot of their end customers. If service based organizations they are making some changes. These enhancements may cause trouble at the end clients. Now they may start raising the production incidents. When this production incidents they popped back to the support team. They have to manually look into the each and every email. They have to figure it out what is the category, what is the type of it? Is this an incident or it's a feedback or it's an enhancement request and then they have to pass it to the respective internal team where QA and dev team they have to look over the incident. Now this is all one manual activity and it has chances of error. Also for example support team, they have done wrong analysis, they are passing to a wrong internal team. Now this support activity is has very small SLA because these are production incidents. Suppose if it is a P1, you have only three to four hours where you have to reply back. I have provided an innovative solution here where you can use AI ML models like text classification which can read the email content can find it out. Okay, what is the category of it, what type of incident it is? And the second thing is use image classification models. Because these end client emails they come with a logo and this logo you can fit to ML model deep learning ML models like Keras TensorFlow library. So what happens? This utility is having two part one is the text classification which is providing you which client it is originating, what is the category. And the second part is image classification which is providing me a confidence that okay, which client we are talking about clubbing both of these output. If you are getting the same client, your automation utility is giving you the Correct output. And if you see what is the advantage of it, we have drastically reduced the manual efforts which were required at the support team and later QA was required to tries the incidents manually. So this one automation utility is like very helpful for several team members.

So I love this because a lot of times people just focus on functional automation and this sounds like an area where you had a lot of manual things and like hey, we can automate this. So a lot of times people don't get involved in that business or find those opportunities or speak up, how did you find it? And then speak up and say hey, actually we can automate this and how did you get the.

Okay, yeah, so actually it was being brought to my attention by one of my senior director. He said like this is what happening and support team is required to invest a lot of time. Can we automate it? And when I have started looking over it, I thought it's a simple utility. Emails are coming. There are some open source Python libraries like Beautiful Soup. You have to just parse the text, find the client, pass it. The problem was like for example Walmart, Walmart is having headquarter in USA as well as in Canada. Okay, from the email text they say only Walmart. How you will differentiate that? Okay, this email is for Walmart USA customer or from Walmart Canada customer from Naked Eye. You cannot differentiate the logo because the logo has same color from Naked Eye. But if you use Python libraries it's a different. And that's where I've used the AI model image classification to figure it out whether this like Walmart USA or Walmart Canada. So like this was an awesome utility. Yes, it was being brought to me as an automation task, but later it turned out to be automation plus AI.

And just so folks know, we'll have a link to both these articles down below to take a deeper dive. All right, so all right, how did AI get involved then? How much do you need to know AI? Because a lot of people are like I know automation, I know no AI. I have to pay for it usually because it's AI. Like how did that. You talk a little bit about more about the AI piece. What is it, what did it cost and how hard was it to get up to speed with it?

Yeah, so AI like it's everywhere. Everyone is talking about AI. Everyone knows AI. The point is like if you have some piece of work, you are writing some automation utility, it does not mean that you have to use AI. AI is just an assistant. Okay? It can help you to some extent, but it cannot Give you the desired output, what you are looking. So for example like AI is now impacting shift left approach. Okay. Earlier like we QAs they were part of traditional testing, okay where they were not getting enough visibility, they were being included only in the dev phase and when the developers they are done with their code then they were starting their testing. Now with the shift left approach QAs they are getting enough visibility, they are starting from the very beginning with AI. Also included with the shift left approach, the locators what QA was trying to find manually, now they can find it through AI intelligent tools like Tosca Vision AI or applitools. So what I'm trying to say you don't need to like start thinking like from the day one that okay, you want to leverage AI. No, just go with the flow. Start writing the automation utility. Just think what are the tasks and if you are finding any opportunity for the AI in between where ML models can ease your work, then you can implement them.

So I guess where do you find these models though? Are they paid solutions? Like if someone's like okay, I probably use AI, I mean how do they know even know where to look?

I guess no, it's not like these models, they are always paid. There are so many open source models readily available. For example for text classification we have natural language processing and there are several libraries which are like freely available. You can use them and you can write an ML model. Similarly for image classification, Keras TensorFlow libraries, they are freely available, you can use them. So it's not like you have to always go for paid solutions. They are open source solutions which are very easily available and there you can refer Kaggle, it's a competition website. So some recent examples you can see where you will find how to use such kind of models.

Love it, I know. Also like so when you're talking about emails I assume like there's a lot of different formats, a lot of different brands, they're constantly changing how hard was it to create it so that the automation is going to be reliable? You don't have to keep maintaining it.

Yeah, so that's a good question. And like whenever you are using AI you have to keep retrain your ML models. The reason is like you were being given one particular data set initially. Like I have trained my this automation email categorization utility on 30 plus clients but over the time what happened like Walmart I have given you an example of USA Canada, Walmart Mexico is also included now and Walmart Mexico is sending emails in Spanish language. But you have not Considered it before. Okay, so your ML model will fail because it is not able to understand that language. Okay, so the answer is retraining. Keep looking why ML model is failing. Then retrain your ML models and deploy them again. And that's how you can increase the efficiency accuracy of your ML model.

All right, so based on the work you have to do to keep it up to date compared to what it was done manually, was it worth automating?

Yes, in the long run, yeah, it is. Because reading the emails manually. Suppose I tell you like 1000 emails you are getting, you are spending several hours, you have to first read the manual, read the email manually, then analyze it. Okay, what is the category of it, which team you have to send it? Now you have all automated. The automation utility is giving you the output. Please direct it to this particular team. This is the issue. It's an incident feedback announcement. What should be the category of it? Is it a P1, P2, P3? All this information is handy with you. The no manual effort is needed.

Nice. So just to make sure it is performing correctly, how do you know? Do you use any like metrics or KPIs that evaluated over time to say how's it doing? Or even to show your manager, hey look, I save X amount of time this quarter, so give me a raise.

Yeah, so automation like to depict your automation efforts, it's like generating the KPIs. Okay, like defects by category or like how much time manual versus automation efforts. So yes, created a dashboard to depict all this easiness as well as how much time has been saved out of this automation utility.

Love it. All right, so you also mentioned Shift left. Shift left's been a concept around for a while. How does AI fit in now to Shift left testing?

Yeah, Shift left, it has been a decade old and it's a boom. Like most of the organizations, they have found value in it. And like during the Shift left, QA is now being involved right from the requirement phase. So QA is having the visibility and as soon as like the product owners, they give you the whiteboard diagrams, mockups, QA start writing the test scripts and parallelly developer is writing their dev script. Now what happens when the developer they are being done, they have deployed their code, they will provide you the actual URL and QA will update the their test script. There is still manual effort needed because now QA has to update the locators for this actual UI with AI in picture. There are some intelligent automation tools available whom you feed this mockers or whiteboard Diagrams on the fly, they will create this locators and tomorrow when you have actual URL available, just change that URL. No need to update the locators. So it is reducing the manual efforts required by the qa. Secondly, we are ready with the automation script. Like once the developer they are being done with their task. QA is also ready with their automation script. They can find the defects and the dev. They can now work parallelly on working on rectifying this code. Other big advantages if we look on a bigger picture, a CI CD pipeline is running. Whenever developer is raising a pr they are behind the scenes, they are running their automation test scripts. If automation scripts are failing this AI tools they can on the fly, they can update the locators and make sure we are seeing more green dots in the CICD tool. So there are less failures because of flakiness of the test. So overall this AI tool helps in like parallel work. Secondly, like cicd, like it's helping us reducing the time and tries and looking over the why the pipeline is failing. These are some key benefits. And one last benefit is QA is gaining more visibility. QA is now collaborating with the project managers, product owners, developers right from the very beginning, like from the requirement phase.

Love it. All right, so once again, is this an open source solution you created like a framework or is it a framework that includes paid solutions?

So for this like when I have used, I've used a paid automation tool and with that paid automation tool, this AI feature was free, it was open source.

All right, so let me make sure I understood the first part. It sounded like you had an image like a Figma image or even like a UI ux. It's able to read the image. So you can start testing before actually having or at least have your test in place ready to go. Your test case before the actual application is developed. Did I hear that correctly?

That is a hundred percent correct. Not only like Figma diagram, even if they are drawing some whiteboard diagrams, some discussions are happening. You just take a photo, it's an image and just feed it to AI automation tools. And these tools, they will find the locators, they will like help you in creating the automation test script. And these locators, they act like ID in Selenium. So they are quite stable. They work on cross browser, cross platform. That's why the CI CD pipeline is also working fine.

What's the solution you're using for that?

What's the solution we are using? Like I have worked on vision AI.

Tool so it's an open source vision AI tool.

Yeah, I've used Vision AI but I have read there are some other tools available as well, like appli tools. So in market there are several tools available.

Nice, nice. Very cool. So. So could you also use it before they develop to look at a UI ux to say hey, this is not a good flow or is that too general?

No, this is feasible only after AI. AI has come into the picture. These automation tools, they have announced their features, they have included AI and now with this AI, basically they are performing image classification, they are reading the images, parsing it, generating the locators for it. All this is possible after AI has come.

All right, so just speaking with other testers, they seem to be getting a lot of either resistance to implementing a new testing approach or there's certain testers is like AI is garbage. It's nah. But obviously you have experience with automation for 18 years and you're creating automation solutions that are not just paid solutions, those also open source. What do you say to people that think that?

I would say like AI is not garbage and it is not a complete solution. Just consider it as an assistant. But ultimately it's you who are going to drive for a robust production delivery. So AI can help you in enhancing your test suit or making sure your tests they are more stable. But it's you who have to apply your brain to make sure your scrap apps they are more robust and there is less flakiness. So you need to drive your automation test.

Nice. So I know as an automation engineer, a lot of times time is spent triaging failing tests. I think I heard you say this, but maybe I'm wrong. So do you have a solution that actually is able to analyze failed test results and triage them automatically? Kind of like the emails, it's able to bucket things. Real issue, not an issue. They're all due to this one issue fix that one issue fix 99% of the test type.

Deal. No, I have not worked on such kind of thing. Like if issues are there then and probably like if they are production issues, then it's best to look manually only rather than relying on some tool whether they are using AI or not. But to be honest, I have not worked on any such utility as of now.

Nice. So do you have any like stories before the shift left with AI and.

Then now like yeah, like before shift left what was happening? Like qa like okay, in the Sprint discussions were happening, there were whiteboard diagrams, mockup diagrams, they were being provided. But I have to write all those tests manually and even if I am writing the automation test script like I cannot provide the ID locators because I cannot inspect the element. It's just an image. When the developers they are done with their testing, it's like they have provided the URL. At that time I am inspecting the HTML elements and updating my automation script script. So this piece of manual effort was still there. The second challenge with automation script is like when you are running on Chrome browser it's working fine, but on other browser it's failing because it's not honoring the same locator. Now with the AI tools in the picture, like when you feed the images to automation tool, they are actually generating locators which are working even when actual UI is being provided they intelligently from the bank if required. This locators needs to be updated. They are doing it themselves. So automation tests, they are not failing because of locator issue. My role as a QA is still there to validate like data validations. Okay, that part is still there, but UI part like there is a lot of easiness because of this AI tools locator issue. I am not facing it any longer. So that's the main benefit in shift left approach. What I have noticed.

All right, so today tester is worried because like, oh, this is going to take my job because I spend 99% of my time on flaky locators. It's kept me employed all this time. What do you say to that?

No, I don't think so. AI is not going to take any job. As again I'm saying like AI is just an assistant, but it's you who have to drive because AI is not having a brain. You are the person who has to write the test per your project requirement and AI can just help to enhance it to make it better.

All right, so you seem like someone that's always messing around with things. What's on your roadmap? Are you playing with anything else with AI that you see could be helpful that you haven't implemented yet?

Yeah. So right now like a lot of things are there. Like production tickets are coming and these are not bugs, they are just tickets. For example, like customers, they are requesting for something. So how can we right now, like we have to look at manually, we have to look what kind of solution we can provide. So my roadmap here is this is one utility where I'm trying to utilize ML models to figure it out. Like when the ticket is coming it can furnish me some information. But based on the previous already Popped up tickets and the second part is like it's again like sort of email categorization. But right now one team, they have some emails, they are using one extension in their automation tool which can read the emails and now from these emails need to label it. So data labeling like it's a different ML model and you need to label. Okay, this email is about which client. So this is a completely a new area for me. So I am exploring it how to label the data using AI.

Nice. So you know, since Genai came out, AI has been around forever. But since Genai came out like two years ago, it's rapidly evolving. It's a hard question to ask but where do you see things going with automation? Maybe in three to five years automation.

It'S like a race and like it's not only about automation. I see like all the fields they are going to be benefited with AI. Specifically with the automation it will be automation plus AI. For example, the utility, what I have mentioned, email categorization using AI. So it, this is an automation utility which won't have been possible without AI. So in the future, in the three, five years I would say like there will be lot of scope, we will be able to use the different kind of AI models which are coming along with our automation utility to reduce the manual effort. For example in the healthcare domain like I can see I have read like a lot of new ML models are coming up like Clinical bird. So that field is rapidly increasing. Doctors, they are trying to use AI there. So along with the automation this utilizing AI models it will be very helpful.

Awesome. So if someone's listening, where do you think they could get started if they want to dip their toe into like the open source AI thing to help them with automation Is that one library you say oh you got to try this, you probably can get benefit right away.

My favorite one is nlp. I'm a big fan of text classification. So I would say start with the natural language processing and the easiest example you can start is like, like looking over restaurant reviews. So restaurant reviews. It's like it could be structured text, unstructured text and how NLP model can give you the output. Okay, this is a positive review, this is a negative review. That's the easiest way to start with and it's very interesting. Some libraries you can use like column transformer libraries here then like spacy you can use.

Nice. So there are any other uses that you see shift left or being not done correctly or things that people mess up?

Yeah, that is correct. Like if it's a complex ui. For example, like when you are discussing the whiteboard diagrams and it's a messy diagram or the requirements, they are changing quite frequently. Like in Selenium. Selenium has always struggled with a dynamic ui. Same issues. It's there with the shift left approach. Okay. AI cannot help you in that. So again, like you need to invest time manually.

Absolutely. I've seen another trend with people moving from Selenium to Playwright. A little off topic. What are your thoughts on that?

No, that is 100% correct. Selenium, like you require works from the back end, but Playwright is rightly on the ui. It's quite fast. And the scripts like I have seen the performance of Playwright. It's much better than Selenium.

Controversial. Nice. So do you see any implementations for these libraries, maybe to incorporate Playwright with some of these AI modules?

Yeah, yeah. So ML model is completely like a different program. You have to just invoke it from your automation utility. So what happens? Like I give you an example of AWS architectures. Like suppose you have written a lambda function and that lambda function is performing several tasks. You can keep your ML model in a S3 bucket as a pickle file and just invoke it in your lambda function. So your lambda code is there, which could be like you are saying about Playwright code. So that code has to just invoke your ML model. Okay. So even in the Playwright you like, whenever we deploy any piece of code, it could be in jar war, this kind of format. Similarly, ML model, once it is working, you have to deploy it in pickle file format. To invoke that pickle file, you can just call one API. Okay? That API will take the input, invoke the S3, invoke the lambda function. Lambda will invoke S3 will give you the output back to your Playwright code. So it is very doable, it is quite easy.

Very cool. Okay, Guarav, before we go, is there one piece of actual advice you can give to someone to help them with their AI testing efforts? And what's the best way to find or contact you?

Contact me. So it's my email address and like on Twitter GauravMittal 1985, you can always reach out to me and I would be very happy to help in any kind of automation task involving AI or non AI. So I would just say please just go with the flow. And like if AI is required in the automation, you will get stuck. And when you will Google it, you will yourself find a solution. And AI models, most of them like till now what I have encountered they all are free, they are open source. So don't worry. You have to just write some piece of code for your ML model, deploy it and in your actual code just invoke it. It's quite simple. Like two different kind of programs, they are interacting with each other.

So do you google it or chatgpt it now?

No, I google it mostly I google it.

Really?

Yeah.

Nice. Thanks again for your automation awesomeness. The links to everything of value we covered in this episode head on over to testguild.com a523 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. Joe My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges. We're excited to to collaborate. Visit testguild.info to explore how we can create transformative experiences. Together. Let's push the boundaries of what we can achieve.

With lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing us secrets to behold.