Maintaining your test automation with Trisha Chetani

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. And today we'll be talking with Tricia all about maintaining your test automation. Tricia is a software test automation enthusiastic. She's been helping her team to follow testing processes to support and enable teams to deliver high quality software in a DevOps environment. She's also really enthusiastic to attend conferences, meetups for professional development. As well as being an active community member. Tricia also loves to work and has begun to think about the big picture. So she's looking for opportunities to thrive in a management role. So if you like what you hear, definitely check her out in the show notes. And if you have a role for her as a management position, I'm sure she'd enjoy that as well. So really excited to have Tricia here. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSource Labs and click on the exclusive sponsors section to try it for free today. Check it out.

Hey Trisha. Welcome to the Guild.

Hi Joe. Thank you. Thanks for inviting me and giving me this opportunity.

Awesome to have you. Before we get into it, is there anything in your bio that I missed that you want the Guild to know more about?

No, I'm good.

Cool. All right. So what caught my eye is I believe you're writing an article on how to maintain or update your automated scripts and I think it's something that a lot of teams struggle with. So I'm just curious to know why you decide to focus in on this article on automation code maintenance. Why is it so important to you?

It's just like I've been approached by Agility Test and they wanted me to write article on this theme. That's why I've written on this and contributed on this part.

Nice. So I assume in your company that you work for now you encounter issues for maintaining automation. Can you maybe tell me a little bit about what that's like in my.

Company or in my current project? There are a lot of issues or challenges with that test automation. It's because there was no requirement defined and we were just doing our testing in the application. We have been given with the application and we were just doing our test like testing and when we did exploratory testing we got quite significant results because we end up finding 700 bucks. But due to there was a lack of what do I say? Requirement, there was no high quality test cases or no high quality test automation. Because of this we cannot say like we our test coverage is good or like our test automation is good. So I would say those were the challenges.

Awesome. I think it's a common challenge a lot of companies face. I think ways how did you get out of that to make it so. So that I guess you don't have so many tests to maintain.

So to get over this, what we did is like every, like every now and then there will be a release and we actually stated that as a console saying that there's no requirement. And for current release, whatever requirement we are getting, we are actually like writing our appropriate test cases and appropriate test automation. But something which was there that cannot be like achieved. And whatever is there, whatever is existing, we are trying to maintain it. But whatever new which is coming, we are actually trying to improve on that aspect.

Great. So I know one point a lot of people struggle with is keeping their test code in a test book edition. So is that something you've seen as well? How do you have Sprint teams make sure that they don't overlook their existing test to make sure that they're still not breaking them each release? You know, know if that makes sense.

Yeah, it's quite challenging because every time we go to different aspect of the project, sometimes it would be like we are just completely focusing on estimation, sometimes we are focusing on requirement defining, sometimes we are actually upskilling ourselves. So it's very hard for us. Or sometimes a new joiner is coming and he's contributing to the test automation code. So it becomes really hard to maintain the test automation code at the same pace, at the same level of. What do I say? At the same level of the code which was there or which was expected by the team members. It's really hard because a lot of dynamics around the project will keep changing. So that's why the maintenance and certain rules around that, like it actually helps the team member who were joining new or whatever. The aspect of project is that if we have a rule for ourselves then we can actually keep on maintaining that. And like for our team, what we thought is like it should be, the regression testing should be equivalent important as user story testing which we do in the Sprint.

So I definitely agree. But how do you get your team to agree to do that because it sounds good but actually in practice I've seen teams like it's too much work.

I completely agree on that aspect. So it's like we actually told it to our management and they actually helped us to have one day as an operational excellence day and certain conditions and certain condition. It was like when test code automation is actually reducing to certain percentage the failure then they were actually giving us. So it was like how we are actually asking the help of the management and then they are giving us the time to maintain that because after talking to them or after having so many conversation with them, I really understand they also like to have this regression through it and they also wanted us to contribute on that. But they don't want it to put an extra. Like we are just keep on maintaining that wherein we are not releasing that it will become hard for them. So that's what they.

So of those failures, how much of them are real bugs? Or is it poorly written test like. Or is it an environment? Like what are the normal buckets of failures that you normally see?

I've seen a lot of time real failures also and sometimes like a developer have actually changed the HTTP URL to HTTPs and the whole test suite was failing because of other HTTPs and my automation test suit requires SSL certificate. Then recently I saw like the failure because we were not able to run the test on what do I say? The selenium grid. Because the grid was updated and our test was actually like using the older version. So I've seen most of the time a real issues also at the configuration or with anything wherein the whole test suit will fail or sometimes anybody have changed the developer have changed the features and we are seeing the real failures with those test automation. But sometimes our test automation gives also the fake result.

So I guess it depends on each company. So I'm just curious to know when a test fails, who's responsible for fixing it? Do you have like a test team? Is it a sprint team? Whoever's in that sprint team, is it one person like who's assigned to actually fixing failures.

So in my current company, what we do is like as you mentioned earlier, it's completely true that different company, different project, different people have different aspects of handling or tacting the things. In my current project, what we do is like as we see the lower percentage or we see like we have a regression cycle because we actually change ourselves to actually adapting ourselves to a brand strategy and branch strategy. We have like we run that test in the feature branch, then we run the test again on the Common SID and then we go to uat. So when we run the test on common SID or sid, which is our testing branch, when we run all the code is merging. That's where like couple of testers will actually make sure that our pipeline is going green and we will make sure from whatever from feature branch is coming to testing branch it should like the pipeline is going green, green, green.

So that's great. So another thing that just popped into my head that I can imagine is an issue is test environments. Because you're running on multiple test environments, how do you know a test if it fails is not due to because the data is not in that particular environment? I don't know if you've seen any of those type of issues.

We are seeing a lot of issues about that. But in my experience what we did is to avoid this in my previous company is to actually have a test environment like automated test environment where like we are actually pre populating data. So when we are before running a test so when we have those pre populated control data then we know like I need to go and select the values in the drop down which is in the first or in the second it's not going to the hundred and the drop down. So it's much controllable. So like it should be cleaned every time there is some maintenance of the environment we need to do it in order to maintain that flakiness.

So when a test does fail, is there anything you can do to make it so it's easier to debug or fix?

Yes, I did a lot of thing around that. I added a screenshot, I added a logs and our test automation or the framework like Selenium or the API framework like Rest assured is giving me our blogs or I can also get a logs from application which is under test or I can for the UI application we can have a JavaScript logs. So those were actually really helping me and like sometimes if there is an uncertainty I'll add a retry logic or something like that. So which helps me to give that more reliable results.

What I've also seen just in my experience working with a lot of people is just even the way you write your test, it's hard to understand, it's hard to maintain or fix and different teams write tests differently. Do you have any standards that help maybe avoid that issue?

I completely agree with this challenge that different team is up writing a test differently or different people are approaching differently. So like and it's. It's really hard to control each and every testers like where you have written in certain way, why you have not written in certain ways. But when I'm doing a code review will actually make sure that the basic, certain guidelines are like, are there in the framework and if they are tweaking little bit, I don't really care about that because sometimes people need autonomy so that they can actually bring more value if they don't feel that autonomy and like it really becomes hard for them. Like if we are actually making it, sure. Like you don't need to define this variable over here, you need to define it this variable over there. It becomes really hard for me as a code reviewer also. And for a person who is contributing, it becomes really challenging because they don't have autonomy. But when I'm doing a code review, I'll make sure like certain basic guidelines which we have developed the framework and those guidelines are being added and it's not breaking or it's not failing all the test keys which with their new pull request.

So you mentioned code reviews. Is that a requirement then? Before someone can up check in their test code, it has to be code reviewed. Do you have some sort of process in place for that?

Yeah, definitely. Like whenever me as a tester and the team who have actually written a new test cases, I'll actually write, I'll create a pull request. And after creating a pull request, like when I've created a new branch, there is a like set and GitHub or a big bucket wherein it will read on the whole test and once it rerun the whole test in that environment, then I could actually create a pull request. Once I've created a pull request, people, I'll assign it to the people. I'll write the appropriate comment on that, what this test is doing and then I'll ask the reviewers to review my code and once they have reviewed, I'll incorporate their comments and suggestion and then it goes to the main pipe, main master branch.

Cool. So how do you get the team to help though? I know. So you're doing code reviews, which is good. How do you like create like a whole team kind of approach where everyone's collaborating and really not just like, oh, they're just flaky tests, forget about them. Where they're like, oh, this matters, we need to fix this. Is there anything culturally you can do to help get your team more involved or more active?

I think if I have to get the help from a developer and one of my experience, what is like we actually spoke in retrospective and then we actually reduced developing of features actively in the Sprint, wherein We had more time for operational excellence day. And like I actually find a pattern over there saying that. Let's see, there was a homepage wherein the database was getting a lot of numbers from different database and that homepage was taking a lot of time for us to actually like display that numbers in the webpage and for me to actually calculate via API and then calling a UI to check that particular number. And the test case was failing. So I actually seek the help from a developer and then they actually optimized the or what do I say page load time. Similarly, once it happened with the what they are saying like they were having a load balancer wherein like when the request is going to one load balancer it was working fine. But when the request is going to the second load balancer I was not able to see the data in the front end application and my test was failing. And this was really hard to reproduce and explain it to my developer. But due to the thing which I had, the log, screenshots, everything, it was much easier. It became easier and I could actually help to show to the developer that certain time this is happening and certain time the test is passing. So they helped me. After analyzing seeing the application log, I could say one more example saying that when we were actually using Docker and the docker was each time when we are setting up application. So what was happening is like it was creating a new image or new what do I say? And that's where it's. It was actually preoccupied in the sys server and it was occupying a lot of memory and there was no process to actually clean up those Docker image and things like that. And when it was occupying a lot of memory in the server, I cannot boot up the application to run the test. I. And that's where like the whole test was failing. So those were the several instance wherein like I could get the help from a developer and they were actually collaborating it. But after all this like I've. I've asked the help and they started having those enthusiasm to actually help me and they what. What I found is like the developer became much more interested and they started like looking at my code, seeing what is a coverage and they were giving me suggestion could you actually like write this kind of test cases which could help me. And that's where we started collaborating more and then they actually feel more enthusiasm and they actually like started contributing with me like doing a test case review and then they started rating up actual test cases because they started seeing the test results going green, green, green, green. And it Was not like a red, green, red, green, red, green. So that's where it became a real team collaboration. And for this everybody was very happy like because developer were actually became more enthusiast to write that test case and like people were really happy because usually it doesn't happen. But it long time to get that trust and long time to show them no tests are not flaky. It works. We need to write it in a way wherein it will work.

Absolutely. So how do you justify a test that's failing a lot though? Rather than just delete it, do you ever just say like this test, even if it fails a pass, like why are we even running this thing? Do you ever get into those type of discussions?

Yeah, definitely. I've undergone this kind of conditions a lot of time and I've also felt certain time like when I come new to the project and I really don't understand about application. I would say why this test is there and what value it is bringing. I cannot really understand it. Even though I'm running it, I really cannot understand it. So can I, can I delete this? I requested to my manager and then removed it.

Cool. Yeah, because I totally agree. Why? I think over time what happens, I don't know if you see this over a year or two, you end up having tests that maybe they don't make sense anymore. So I don't know if it makes sense to almost every cord or have like a pruning session where you just start deleting things because they just don't make sense anymore. But for some reason people really don't like to do that. They're like, oh, we need that test without being able to know, like you said justify what the heck is it doing?

No, it really like as per the context, we need to adapt ourselves. And that's why revisiting our test case and automating or updating our test framework or updating our test cases in auto test automation makes Sense than maintaining 5,000 test cases or thousand test cases. It's a real good practice, I would say.

Do you actually have that as part of your process that you do clean up?

Like currently new people are there, so if they don't find that test cases which is making sense to them, and I'm the only old person who is working in the same project for two years, so I know it. So whenever I review it and I'll feel okay, they're saying it doesn't make sense for them and it also doesn't make sense for me. Then we'll remove it because People keep on changing and the application is growing, growing, growing.

How do you handle though? Say you have a test run and you have a lot of failures, but you know code's gonna be checked in soon and you need to figure out why it failed before it runs again or people are gonna start ignoring it. Do you have a way to identify tests that are failing quickly or anything like that so you can help triage the tests fast?

Like, I really don't have anything as such in the process, I would say as of now, but I think it's a really good idea to have it fast. But currently we have a lot of tests at the moment and a lot of things fail. So we actually wait for, we keep on running it and we keep on fixing it. So it's like at a moment any tester will focus on one particular test case, not the whole test suit. So if they have fixed one test cases, then they will look into the another test case. By the time if somebody is changing it, they'll rerun that test and see whether it's passing or failing. But I really think it's a good idea to quick check. But for that, if we have a small test suit or very focused or dedicated, then it's, it's really helpful. Wherein people have lot of knowledge, they are existing in the system for a lot of time, so they really know which, which thing where to change and make it pass. But currently I don't think so. We have that like knowledge or even time or even speed in our team.

So if that's the case, do you have a way to run tests in parallel so that they run faster? I don't know how long it takes, but do you have like a. Like it needs to run in under 15 minutes. So we need all these environments running or anything like that?

Yeah, we have a parallelism and test like I would say we are using selenium grid and we have like test running.

Nice. So I guess we went over a few things already. In order to make tests more maintainable, then you have to specify a goal or what your expectations are for the team. Can you talk a little bit about risk where risk falls into this? As we talked about, sometimes we write tests that have no value for new tests. You always try to associate it with what's the risk area. Why are we writing this test so that you don't have that more and more of tests in there that don't make sense.

I think when, when I see the user story in the Sprint, what I do is I'll actually do an impact analysis. I'll say like, what is the priority of this feature? What is a business impact? What is a risk of like not having this test and the regression and that's where I'll think of whether it should be automated or not automated. If I just don't go and do each and everything Automate like by automation I would have certain user story in Sprint wherein I decided I'll not automate it.

So as part of your planning, do you have any rules of thumbs like of what your test automation strategy should be for teams?

I would say currently we don't have a solid test automation strategy across team. But what I would say is I've actually gone through many talks, I've gone through many of talks and blogs and things like that. So that helps me to decide whenever I'll see any new feature or anything saying that this should be automated, this shouldn't be automated. And I do have that experience when I'm like when I'm the Sprint, like I could actually see like falling into the test pyramid also saying that this is, this could be tested via API and we don't need it to do specifically via a UI because currently like I have given that liberty and people are trusting me so I could make those tests and what do I say? More targeted player. So that helps me. And when I'm actually working up the people who are reporting to me, I help them to understand that when I'm actually doing a Sprint planning or things like that. But I would say I've not actually defined a test strategy wherein I could say to everyone this is a test automation strategy and we should automate it like that. But when we have those conversation with anyone, what I explain to you, I'll try to explain them, I'll try to do it with couple of examples or a couple of user stories, but there is no written document and no solid process wherein people are following them.

So you also mentioned test data earlier and you said you have a lot of new people on your team. How do they know what data is needed for them to create? Does that make sense? Like is it?

Yeah. And it's a real challenging topic for them because being me as like a little old person in the team, I know like let's say if I have to change the permission in the application, I could do it over here and that's where I would see it, something in the UI with the changing of permission. Or I would say when I need to have something as invalid data in the application I would know I could go over Here and query in the database and I could get this data here and that's where I could check in the front end. Like if I'm getting those values, how do for this, this particular dependency I'm getting those wrong value. How does the application should react? Does it giving an error message or what what's happening? So it's a little easier for me because I know the context little bit. But for them it's really hard and sometimes it becomes so hard for them that they're not able to write the test cases because they say it's. It's very dependent data dependent and to write each and appropriate steps in the test cases it becomes really hard. And when they don't get that logic of writing a test case itself, that's very hard for them to automate it. Let's see.

So you know I mentioned in your bio that you were interested in becoming more of a management role. And for me, being an old guy, I never wanted to be in management. But they tried to force you into management, so why get into management? Do you see that as a good path for testers to get? Would it be engineering management? Just curious to know why you would think make that career path.

I would see myself when I actually became a tester, I was just knowing a limited part of what do I say my job and I was doing a limited thing. When I became a senior tester, I was knowing little more how to do a backend testing, how to do a performance testing, how to do different kind of thing. Now when I became a test lead, I know little more of the context and I know why people say this. Having this many bugs is good to release for them or I could understand their context or else previously I was like why people are saying we have a blocker bug and they wanted to release it. But now when I see a lot of bigger picture and understand their challenges, understand their priorities so I could make sense of like what I was doing as like I was trying to just help them and how and why. So when I get I take more responsibility and I try to see more impact and I try to see what are the challenges other people have and how could be in my role. I could help them. It helps me to learn more and it helps me to think from a bigger picture. And that's what I've mentioned in bio. Like I wanted to see that bigger picture so that I can learn and I can grow myself.

Nice. This is probably a different podcast episode, but I'd love to know steps people can take in order to transition into a management role. So once you make that leap, I'd love to have you back to talk about that because a few people asked me about it for sure. Okay Trisha, before we go, is there one piece of actual advice you can give to someone to help them with their automation testing efforts and what's the best way to find or contact you?

I would say having a constant reflection on their test automation code is our best way to maintain that. If they're not having a reflection they are not actually trying to maintain it and it may go the risk would be like it may go to somewhere else. No people are maintaining it or no people are actually thinking of how to improve it or how to maintain it, or how to even think of creating a new framework or something like that. So maintenance is hard, but that's super important. I'm pretty open with my Twitter. My twitter link is trashachetani and my email is them. I'm open at LinkedIn so anywhere people can contact me and I'm really helpful to share my experience or even learn from them. They are experienced experience which I was not knowing about.

Thanks again for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com a397 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Labs awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to test guild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.