Speeding Up Automation Analysis with Ruslan Akhmetzianov and Natalia Polyakova

Foreign.

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey, it's Joe and welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Rosalind and Natalia all about speeding up automation analysis. As an automation engineer, I know how hard it is to analyze all results, especially if you have a lot of failures. So you're going to learn how to save time analyzing automation test results in this episode. So you want to stay around all the way to the end if you don't know. Roslyn is a devrel lead at Quemeta Software. He's a Test Ops ambassador, a DevOps enjoyer, the conference crawler is and he's been a guest on the show before, so he brings a lot of knowledge. Really excited about this. We also have Natalia joining us who's been working in the test industry for almost four years. She started as a trainee and she finished as a lead to leave the QA path for automation. So she's the perfect guest for this show. She's now a quality engineer at Quemeta Software and works as a mentor for junior QC engineers at a tech university, which is really cool. And she believes testing should be part of the whole team's work, not just testers. Wholly agree, definitely agree with that. So you want to stay around all the way to the end to listen to all the knowledge we're about to drop here. So let's get into it.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit test guild.comSourceLabs and click on the exclusive Sponsors section to try it for free today. Check it out.

Hey, welcome to the Guild.

Hey.

Hi.

Good to have you. I guess before we get into it, is there anything I missed in either of your BIOS that you want the Guild to know more about? Sometimes I botch them.

No, I guess you did well, so that's awesome.

Cool. All right, great. I guess before we get into the topic then is, you know, a lot of times automation engineers struggle with a lot of things, but the biggest things are results. So just curious to get your input on what you think about that statement. Do you see that as well and what type of test results and things like that. You think a lot of automation engineers struggling with. So either one of you want to start?

Yeah, I believe that Natalia should start because she's actually a quality engineer, so she knows about the test results and so on. And I'm just like, no, I just love to talk. So, Natalia, please.

Thank you, Ruslan. Well, many manual tests to test some products and find a bug, he just creates a bug report so the whole development team can understand what went wrong and how exactly it must work. So the automated tests are not as proactive as manual testers, and frequently they are not able to give the development team clear information about a bug. Developers or testers should read the test results to find out if an automated test failed because of a real bug or because of something else. It can be a bug in the test's code, or it can be a temporary server error and etc.

So, long story short is that tests are actually dumb, so people are usually not. And at this point, we just get the whole story about a lot of new processes and entities popping up as soon as we keep automating stuff and trying to keep this automation under control. So Natalia went the whole path from manual testing to test ops, which we've discussed last time with you, Joe. So I believe she can tell us about that, about these.

Entities and processes which pop up actually to enable actually reading the results. Because when you just use a testing framework in an IDE or on a pipeline, all you get is a huge stack trace. So nobody wants to deal with that. So, Natalia, tell us, what do people actually use now and what's necessary to make automation work?

Yeah, automated test results include failed scenario steps, traces, error messages, logs and attachments. A lot of all. But all of this data can be handled only by testers or developers. And in manual processes, testing is time consuming and in automation, tests run fast, but the number of resulting increases and analyzing everything may take a lot of time at all. So what can we do? Yeah, we fortunately can automate a part of routine work. And of course, it's about regression test and about sanity suits. We can do what, for example, we can handle this flaky test, first of all.

So I just wanted to add that as soon as you get regression and like, these are usually the first steps, like, nobody wants to run the same test. Like, each time, like a feature goes out or a release is coming closer. Like, nobody loves routine except automated tests. They do love them. But at this point, when you get like a thousand of automated tests, which, like, should work for 100 of usual manual scenarios because you want your automated test to be small, atomic, like clear and easy to run and to read and to fix. So but at this point when these tests like pop up, a lot of them and you start running them again and again. So just a couple of weeks ago, I've been talking to people from Q engineers from Miro Team, like a collaboration tool, I guess most people know about that tool. So they run.

3 thousands of tests 400 times a day.

There is like 600 of engineers working on the product and they create somewhere around 200 of pull requests. So they make a run before a pull request and after one. So at this amount of test runs, you definitely get something wrong. Some tests are like, how do you say it? Like they are false negative or they are just unstable. So and speaking of unstable tests is a huge topic.

How do you lose? How do you keep the developers engaged then? Because a lot of times they're like, these tests fail. I have no confidence in them. I'm not looking at these reports or results. I can't trust them.

Yeah. So there is also a point when me and my colleague Khartoum, who is the co founder of Qaemeta, we've started like speaking not about trust, but there is also faith into the automation. So you should be faithful for automation because, you know, trusting is something you need to get proof. But to make your automation like evolve, you need to have faith to believe that it will work. If you put a lot of effort into this, like that's how it works. And as soon as you start getting flaky tests, and like we all know what flaky test is, is a test which like can fail or not fail depending on something that happens under the hood. Like is the test broken or is the code broken? Or maybe it's infrastructure or maybe it's like bad astrology or like bad luck. Nobody knows what happens when a flaky test become flaky. So that's something to discuss because the reasons for such behavior are often not obvious. And there is also like, I believe a lot of testers read like for example, Uber's blog about engineering. So there is a whole team working on flaky testing and they build huge frameworks because like they've got dozens of thousands of tests and they always fail. Like each run something fails and at that point you don't understand, like, is it a bug or is it like a bad time to run tests? Or maybe the machine, like maybe a RAM in a cloud just works inefficiently so. And at this point we have to think about how to save time working with flaky tests because it is time consuming, like investigating something takes a lot of time. So I believe Natalia knows a couple of ways to work with flaky test. So Natalia, please.

Well, frankly speaking, some people say that if you have a flaky test, you should fix it. And there are lots of ways to fix flaky test, but sometimes it's not a good way. And when we want to save time, we can do these two things. As Ruslan said, first of all, we are able to rerun all failed tests. This opportunity is already realized in the most popular frameworks, Junit, Pytest, SNG and etc. We just several times to run the same test and hope that in one of the runs it will be passed. So if we have a flaky test run can help us to understand that occasionally failing is not a real bug. And the second thing that we can do, it's muting or quarantine tests. It always can be called as ignoring. That help us to not pay attention to unstable test reports. And usually muting or quarantine is one of the test framework tools too. So we can find it in junit and in testng always too. We just need some annotations or methods to to make some test results invisible for launches. When we fix this test, we just remove the annotation and then our test will work life like a charm.

Yeah.

Did you have a delete test just before Rosalind, I'm just curious. I know you gave the option of rerun. A lot of people freak out when I tell them to rerun that test. They're like, oh, you're skipping over a real issue. I like quarantine. I found that really effective. But how about deleting? Like do you find teams just maybe create tests for everything that necessarily maybe you don't even need the test and delete it and that would save you some time?

Well, that's a tricky question. I believe. So deleting a test is a good way when you know the test does not work for you. So that's the only case when you should delete it. As far as I believe and like, as far as I understand, like if a test fails and then you delete it, you just like, that's the real ignoring of a test. Like it's not a quarantine. That's like something. This is a way I tend to work with my issues in real life. I just ignore it. Like oh, there is a problem. I can get some sleep. Yeah, so maybe it will just go away, but it won't. And another thing important to point out is actually a failing test is a test we should love because it actually does its job. Because if we got the green pipeline and a lot of bugs, there is something wrong. Right. So analyzing like results is important and especially like focusing on real issues opened by failures is good. But so deleting a test is good when for example, I don't know, like when you actually don't need to test something like I don't know, like feature has been deleted. Of course then we should delete the tests. But deleting a test because it fails, like it's just the same as laying off people who actually work.

And we want to forget about test coverage too. We shouldn't forget about this.

Yeah, and test coverage should just increase all the time, not just it's not going down. Right, right, right, yes. So, and at this point I just wanted to say that these two basic approaches brought by Natalia are like really basic. So there are a lot of complicated and complex solutions to work with flaky tests. But if you are trying to save time and to work efficiently and you are starting to handle these issues, don't go for complexity. Like I don't know if you know, like Venkat Subramanium, well known speaker from United States and consultant and so on. So he has a talk about Java and the talk is called don't go from complexity run. Because complexity is something you don't want to be present in your software pipeline or in your code or anywhere. Because the simpler the solution the better. And at this point this is why we actually worked on implementing these two simple approaches in Allure testostops because as soon as you try to build a complex solution in a pre built system, it won't match. Like it won't fit a lot of teams. So each team has like specific needs and so on. So it's more like it is more about working inside in house software development of tools. So but if you build a platform, you try to build something basic which will work for almost everyone. So this is why UnluteStops supports the functionality of rerunning of tests in frameworks, for example. So but as soon as you start rerunning tests from Junit or Testng or Pytest, you just get this last result and you have to watch for it like it's passed. Okay, should I rerun it? Because it might fail next time? So you rerun it and Then you just, okay, now it still passed. Should I rerun it again? And then when it fails, you just, how, like, what's the percentage of failures? Like, is it a flaky test? Because in some teams, people, like, say, we call a flaky test. A flaky test is a test which fails, like more than 10% of the time. And if it's like 10% of failures, it's more green than red. So we say it like it's an okayish test, but something goes wrong in other teams. Like, just a single failure in the whole history of test runs makes a test flaky. So that's not a topic to discuss. But that's like, really up to the team to believe what a flaky test is. And this is like another issue for tools builders like us, because we can't make a single flaky test function, you know, like a fixed flaky test. Because a flaky test for each team is like, is a separate thing. So what we do is we provide, like, capabilities for people to see the whole history of reruns. If you've got a test case in a launch which is failed, and then it was rerun automatically, for example, by a framework, you just get all these lines of passed, passed, passed, failed, passed, failed, failed. And then you can look at the launches at the time of launches and everything. This is how we support the rerunning tests. And if you got 10,000 of tests, of course you don't want to run 10 times, like 10,000 tests for 10 times. So rerunning is a good function. That's awesome. And the same works for muting. Like a muted test should be muted.

For some time. You should know the scope, the time scope of a test to be muted. Because if you are just muting it and forget about that, something goes wrong, as far as I believe.

Yeah. I was just going to ask Natalia, if you did mute something or quarantine, how do you remember to un quarantine it? In the day when I was doing automation, you had to actually remember, I need to get rid of that tag. Does using something like Alura Test Ops make it easier for you to quarantine and un quarantine things?

Yeah, of course. It's just one button. You can just click on it and.

Oh, cool.

That's all.

Yeah. And so in order for you to not forget about the muted test, you actually get the number of muted tests in each launch in the results. So they are muted, but they're still like, you get like 5,000 of automated tests, like a hundred of manual tests. And like 90% are green, 10%, like say, or 8%, like broken or failed or something. And then there Is like these 2% of muted tests, which are just the gray pixels on your monitor. But you remember that there are muted tests. You just take a look at them and then, oh, yeah, we fixed that.

Nice. So can you do custom tagging as well? So you have like eight Sprint teams and you notice Sprit Team Z has all the tests as quarantined, so they don't have any coverage on check in. Does it give you any type of insight like that?

Well, if I got the question right. So we do support custom fields. Well, we've got like, each test case may have a number of standard or normal tags, like owner, like feature or microservice or something. Like, I don't remember. Like, maybe Natalia remembers the structure of test case. But there is also, like an opportunity to create custom fields which you just name it and use it as you wish. You may create rules for the custom field or just ignore it and use strings as variables there.

And you can integrate them to autotest automated tests too.

Yeah. As an annotation in your code. So you just work with it within your code, and then you get it in the UI for managers or testers. And then you can create like, any dashboards or results or like data slices sliced by these custom fields. So if you want to get all the muted tests in the runs or in the whole test suite or in this specific test plan, it's actually possible.

Nice. So, you know, we mentioned Allura Test Ops a few times, and for people that didn't hear that previous episode, though, and they're like, well, what is Alara Test Ops? I guess what is it in a nutshell, Roselyn, to get people up to speed if they missed the original.

Yeah. Okay, so the allure Test Ops is, I do call it like a DevOps ready testing platform. So it's a tool which was built in the, like, three or four years ago when testing management systems, like, arise. Like, well, they just popped up everywhere. And everyone started using testrail, which is an amazing testing management system. So. But testing management systems are usually focused on manual tests. Like, they provide you an amazing experience with testing scenarios, working with test cases, like putting them in the right order, like clicking everywhere, getting the results, and then you get the beautiful dashboard for all that. But as soon as you want to get an automated test integrated with a test management system, you have to, like, it's simple. Write an Integration for an API. Like test management systems. Like they provide you an API. So like our founders just thought about like how much time people spend writing integrations. So why don't we create the system which is actually already integrated out of the box with huge number of like programming languages, CI cds, issue trackers, like and now also test management systems. Because Alert Stops has an interface and like has whole blocks to work with manual testing. And it is a testing test management system. But when people want to work with automation. Oh, that's already not in a nutshell. Sorry Joe.

I'll have a link to the episode so people can actually listen to the full glory of Alert Test up.

Yeah, but long story short, so you just get alert to Stops out. You just plug it with your testing infrastructure because usually you've got like three frameworks and two cis, one for web, one for backend, one for like integration testing. Then two cis one is like master or well it's main CI and the second is testing CI or something like that. And then you just plug it in. You don't develop anything, you don't maintain anything. That's why Alert Stops was created.

Natalia, I assume you use Alert Test ops in your day to day work. Does it help you? Have you used anything before that and now you're using it and maybe you see, oh, this really has helped. Now developers are really on board or anything like that. Any other benefits that you've seen?

Yeah, of course it helps. When I worked in the previous company, we integrated with seller two Stops first of time. That was first time when I knew about it. And it was really cool experience because we had normal processes for automated tests. And after Routestop's integration we understood that there are really cool tools to analyze test results and to just rerun test results from one place. Know a lot of different repositories. So CI CD page, something very easy to understand for.

Manual testers because I was a manual tester and my colleagues, my team was were manual testers. So yeah, it really helps. And now of course we use Alertus tops too. I test it and I test it with. With help of it.

Right, right. That's cool. So it almost sounds like a collaboration platform as well because all in one location, the whole team go that one location. Like I said, you're not pulling for all these 10 different things and try to say, oh, why? Why did this happen? So I like that. So I guess what are some other approaches then to help speed up maybe automation or analyzing of failures? You know when you mentioned something about a Lot of these tools are made for not necessarily automation. So they have, you know, defects and severity. Those things also can be applied to automation, I guess. What kind of approaches can help speed up automation is my question.

Of course, flaky test analysis is not the only thing of course that can be automated. And about other things. Yeah, it's possible to automate some processes too with tms like Alert Stops, of course and Report Libraries. They very often provide the tools for bug rule detection settings and some tools for easy viewing of different kinds of attachments, test result history and even visual validation, including tools based on AI ML. And perhaps the most useful tool as I can say, it's automated bug detection. Of course sometimes stack trace or error messages do not contain clear information about a bug. But it can be some error messages or stack traces texts. Some texts are with a readable definition. For example, if you have an error that sounds like select selector, some long and scary express couldn't be found. What does it mean for development team? Actually nothing. But if we replace this error with wrong selector or element is not visible. All team can understand what is happening.

So just a quick follow up, Sorry. So if say you had 100 tests and 100 tests fail and say 50 of them failed, does it tell you out of those 50, maybe it's just one error that caused all 50 failures. Does it give you that type of information as well? So when someone comes in in the morning and goes oh my gosh, I need to debug 50 tests, does it just say actually you just need to debug this one issue?

Yeah, defect detections, some rule for defect detection help us to not only text with error, readable error, but also it help us to define that we have only one back and there's a lot of tests fail because of that.

I love that. I think that would save teams a lot of time. Rosalind.

Yeah, that's actually why defects feature in Alert to Stops was one of the first to be developed and marketed because everybody suffers from huge number of failures for one single reason. Like the page loads for a minute and then all the tests fail because they just don't see the locators, they don't see the like buttons, they don't see anything. They don't have eyes actually, so they never see anything but there is nothing at all. Yeah, so at this point having a defect which hides all the failures like grouped by like some nice regular expression. I don't know if I'm like if it's possible to say nice regular expression. But anyway, so Just use the regexp, something simple to group all the tests and hide them in the defect. So the other feature of like, which we love and that's like how it works, we are using dogfooding, so we use our own tools and we love that you can like link the defect with an issue tracker. So developers don't like testers tools, that's the fact. And managers don't like testers tools and developers tools also. So this is why everybody loves Jira, because this is nobody is like nobody loves Jira and everybody loves Jira, you know, that's how it works. Like Jira is an amazing thing in the world because everybody uses it. And there is so many things about to discuss about Jira. Like there is we can make a whole new series about Jira stories, you know. But at this point you want to like put something out of the alert stops to show the defect is being worked on and the link between the defect and the issue works two sides. So you can see the status of the issue in Jira or any other or another issue tracker within alert stops. And then in Jira you can see how many test cases are linked to the.

So this is how defects work in alert stops. And this is a couple of steps further than just providing a human readable mistakes from stack traces. Right.

And also does this do mapping then? So say they fix that defect and because you have that mapping they know, okay, we just need to run these four tests rather than 3,000 tests every four minutes to verify if it worked. Right.

I don't know if I would say that this automated launching on issue closure is more about configuring a pipeline than within the test ops. Like the test ops is more like a monitoring tool. So it provides all the interfaces to run everything and to see everything. But so how it works like by default is as soon as you close the issue, the defect will be closed automatically. So and if the issue does not fix anything, you will just get all these failed tests back in your backlog and you're like, let's reopen the defect maybe.

Gotcha. Right? Right. All right, so it bubbles up. The defect wasn't really fixed. So how about quality gates? You know, a lot of people spend a lot of time in like Jenkins or circleci creating quality gates. What are your thoughts on quality gates? Do you use them? Is it part of your solution? Natalia, any thoughts on quality gates? Automated quality gates?

Yeah. Well, I believe that at this point I will just take the word to speak about quality gates. So there is no such feature by now, but we are thinking about the way how to implement it. So speaking of quality gates, that's actually. Well this topic popped up in my mind because when I've been talking to Miro people, they just developed a plugin for BitBucket which actually is a quality gate. So when you push your commit like or when you merge the branch, so there is an automatic system which looks for non severe failures for example and it says okay, let's deploy and if there are some like a lot of failed tests and these failed tests are severe, it just automatically like so rolls back the push. And this is how I believe quality gates should work because I've been making a talk on a conference recently on quality gates topic. So the funny thing is when you try to google what is quality gates, you know what you get in 2022? You get the definition which sounds like a quality gate is a meeting where all the people in charge meet and go through a checklist. I mean really there is still a lot of articles which highlight quality gates like this. So it should not happen in the world of 2022. Like everyone is remote, nobody wants to make a call with a checklist. So think about automating quality gates. And to automate quality gates you need to gather the results and this is what we are working on.

Nice, very cool. Love to hear that. So almost out of time. But before we go, I always ask this one, this last question is one piece of actual advice you can give to someone. So an actual piece of advice to someone to help them with their automation testing efforts and what's the best way to find or contact you? So Natalia, do you have any actionable advice you give to someone if they're starting off with automation or maybe how to make their automation easier or better?

Well, automated tests can help us to increase the speed of testing. And however, quality assurance can be provided only by people, not by automated tests. And people using the right tools to analyze automated test results can really improve their product and make their customers happier. And that's why Adopter Stops is oriented on dashboards analytics a lot. The main aim of QA is bug prevention and analyzing how to make product quality better. And we can achieve that only by using the results, not just by writing and running the test, but we should analyze them and to understand what can we do to make our product better.

Yeah, as for me, my advice is extremely simple. So there is time and money and you can get more money eventually, but you never get more time. So when you choose tools, approaches and when you are working on something. Aim for saving time, not money.

Love it. Great. Great. So I have all the links to this in the show notes, but if people want to actually learn about we talked a little bit about Allure of Test Ops. If they've never heard of it, how can they learn more about that?

Oh yeah, it's actually a QA meta IO I believe Joe will just add the link somewhere there is a website and there is a GitHub community created for Allure Report actually so.

The repository is for Allure Report but there is a field where you can ask about stops, provide any ideas or questions or anything and all the team of maintainers and me personally are trying to look into all these topics and if there is something interesting we will come for you. We will thank you for the feedback or for the ideas and tell you how we can achieve that.

Thanks again for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com a425 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test guild.info and let's make it happen.