QA to QE with Lagan Khare

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking all about quality engineering and test automation with Logan. Logan is a manager of Quality engineering with over 13 years of experience. She's a big believer in delivering the highest quality software, ensuring her knowledge. I saw a session she gave recently on an online event and thought she'd be a perfect guest for the show to talk about DevOps, automation, QA, QE, all kinds of awesomeness. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com.

And click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey, welcome to the Guild.

Hey Joe, how are you?

Awesome, I guess. Before we get into it, is there anything I missed in your bio that you want the Guild to know more about?

I think you covered it pretty good. I basically work to push our engineering efficiencies across a wide portfolio of products, legacy and cutting edge teams, both.

That's one of the things I love about your background. It seems like it's a real world situation, but it's a huge enterprise, which I love. So I'm really excited to dive into what you did to actually drive the transformation you have at your company. The first transformation, though I think people sometimes get hung up on, is what is quality engineering? Back in the day I used to call things quality assurance. So I guess to start off with what is quality engineering?

So basically, quality engineering is a set of interrelated responsibilities into three different spheres, which is customer advocacy, deficiency, defect mitigation, and quality ops. And to explain this a little bit further, so the QE quality engineers are basically customers advocates. We represent customers. We're in the shoes of the end users. We're trying to test relevant things, making sure the requirements are correct and the product is working as per the expectations. The other sphere is defect mitigation. So we want to ensure that we're testing at every layer. Like starting from there are unit tests, there's service tests, there is front end testing, automation, and then performance engineering and all those good things. So that's the defect mitigation part. And now the most interesting one, and which I feel is Newer to this domain is the Quality ops. So how quality engineering can be integrated with the OPS here. So like it deals with deployment, ownership, environment ownership, quality engineering, it's just more than QE because quality assurance is making sure that, that the product is of a good quality, finding defects. But quality engineering takes more than that. It talks about getting involved in the overall process of engineering, right from requirements gathering into build process into cicd, into releasing the software to production monitoring and in collaboration with different disciplines across the entire sdlc.

Nice. So I guess a lot of people might then ask how's this different than DevOps or Agile? Is it like one in the same? Is it something different?

So I think it's definitely different than Agile and DevOps, but I also feel like modern day quality engineering, even though it is not the same thing, but we have to work in collaboration with these different disciplines because quality engineering on their own I think can only go so far if you're not collaborating with our dev partners, our partners in delivery management, our partners in the DevOps world. So it is different but I say we should work towards bridging that gap. So like some of the things that we've done to bridge that gap is into introduce like squad based testing where we expect the entire team to help us with testing to make sure that there is no like QE doesn't become the bottleneck and there is a continuous flow of testing throughout the sprints. And then at the same time we're also working with the DevOps team, monitoring and synthetics and making sure that QEs are collaborating with them and helping out driving those network synthetics. Because as a quality engineer I think we know the product and what areas are most and where networking should be there, what kind of alerts should be set there. So we're working with DevOps to sort of in that collaboration to provide that insight into what is important, what should be monitored and how to sort of keep those tests updated. And then similarly with developers like we had a goal last year, we call it find it fix it Again in the terms of bridging the gap between QE and dev, where we literally asked our QE engineers to find a defect and if you know your product well, if you know the tech stack, submit a PR for that defect fix.

Yeah, I love the concept of find it fix it. You did mention something about squad based testing that just got me thinking, how does this work in your organization? Do you have sprint teams with testers embedded in the sprint teams? Is it still a separate team but then you all get together, do squad based testing. Tell me a little bit more about the process I guess of getting QE involved in your teams.

Ye yes, you're right. We have different squads in our organization and QEs are embedded in those squads. And since we all know like the deaf QE ratio is not always one is to one. And sometimes if we don't want testing to become like a bottleneck. So like if there's one QE on a squad and they're not available, we don't want the testing to stop. So again QE would still be there to make sure that the quality checks are there, the gates are there. And we then train the developers to help us with testing so we can start with manual testing. Devs can also contribute to automation testing. So for squad based testing, the concept is to have everybody involved in the squad contribute to test because that would avoid QEs being the bottleneck. And then once the stories start getting stack up in the testing column, usually they decide like a work in progress limit. So if there are more than six or seven tickets in testing, then the developer, one of the developer would pick up a testing ticket from in testing column and help out with testing rather than picking up another ticket for development. And the only rule is though a developer cannot test their own story, so they can help out testing the stories and tickets developed by the other developers. Developers can also contribute to the automation and regression testing or release testing. And it then depends like team to team, what the team dynamics is and how they've understood it and how what's comfortable for each team.

Awesome. So I think in your presentation you had like squad based testing underneath, bridging the gap between QE and dev. You also had something called about, I think it was have a reliable feedback framework. So I guess how do you have a reliable feedback framework? How does that, how did you put that in place?

So for the feedback framework we wanted to have feedback at every level. So like it could mean two things, right? Feedback on the product and then feedback on also on the quality engineering efforts and like what kind of testing we're putting into. So we address feedback on different levels for the product. We want to make sure that the developers when they're testing, they know how to give the feedback on the stories that they're testing. There are rules in place. There is a proper test case management system, there are proper channels where they could get the feedback back to the original developer or the quality engineers about a particular ticket like from feedback framework regarding how the quality engineering is doing. We want to know what if the developers and the other team members, they know about, they understand like where we are in terms of testing? Like how transparent is your quality engineering? Do you know what goes in your smoke test? Do you know what regression comprises of? What do we do for production validation testing? So we want to make sure that there is a channel to sort of get all that feedback back to the quality engineering group.

Very cool. So you had another cool slide here on measuring how quality engineering is done. Now you mentioned surveys, I believe in the session. So is this surveys your company has done? Is it what other companies have done? What are those metrics that you found really helpful to actually measuring the QE transformation, how your teams are doing?

Right, so we're actually measuring eight different metrics to sort of measure the engineering efficiencies. So the four are the standard DORA metrics, the DevOps research and assessment metrics. So that comprises of lead time, which is basically the time a ticket takes to in business days to take, the time it takes from code to commit to live in production. The second one is deployment frequency. We measure these metrics monthly, so how often we deploy to production in a month. Then another one is change failure rate, like what percentage of our releases are successful versus which changes need to be rolled back. The fourth one is meantime to recover. Those are the four DORA metrics. And then additionally we track four more metrics which is a rework, production release activity time, critical automation percentage and automation runtime. So rework is basically so in your workflow if anything goes back backwards. So like if a story is being tested and you find a defect and it goes back to development, then and for any reason, it could be config issues, it could be build issues, it could be defect. But if there is rework, we count it as we do count it because we want to look into what why there is so much rework. And these become like really interesting topics for retro discussions or team meetings where we find areas for improvement. Like for example, for a team of the rework is really high, say 25, 30, 35%. Then are the acceptance criteria not clear? Are developers and testers talking to each other? Is there enough clarity on the stories? Is there enough context around the stories? Are we doing refinements properly? So these metrics are really good indicators of sort of figuring out like what, what's a problem area, then trying to resolve it. Production release activity time is basically does it take for a team to release and release to production and do the prod validation? We want to keep this number low. If we want to increase the deployment frequency because if your deployments take hours to push code out to production, definitely we don't want to do it again and again. And generally there's an understanding that people don't want to like spend so much time in releases, which is the opposite of where we want to be. So that's again a really good metric to measure and track. Then critical automation is basically what percentage of your smoke test suite is automated. We want this to be 100% at least because we're not tracking regression here because we want to be real careful about what we want to automate and how much we want to automate because it just then becomes a big maintenance activity if we automate everything without prioritizing. As much as I love automation, I want to be careful with the automate and test that we're not going to run again and again. We don't want to spend too much time there. So that's why we're tracking critical automation here and then the automation runtime because it speaks volumes about the teams, like if their critical automation runtimes is in minutes, is in hours, how long does it take? So if it takes two to three hours then it means that there scope of improvement, where are we running the test in parallel? Do we need to scale the infrastructure for testing? So these are some areas which gives us insights into like what's the state of the team and what are the improvement areas that we can work on.

Nice. So you touched a little bit on test automation but we haven't actually touched it in detail here. So just curious to know what is your high level test automation strategy? And then I thought maybe we dive into maybe certain areas.

Sure. So for test automation I believe that there has to be. So we don't want to sort of create this big backlog of automation where we are always automating after the sprint is over, after the story is released. So I'm a big proponent of considering automation in the definition of done. Now automation should be done in different layers. So there should be service level automation, there should be automation for the ui front end stories and tickets. But again like there should be a careful discussion between the QE teams and the stakeholders. Like what is a P1 test case? What is a critical path scenario that at least that should be automated, which are our stable tests, which tests are likely to change, you know, but we should have some sort of agreement and buy in from the entire team that this is something that the test team is going to invest time in and we don't want to push automation to the last step or we do automation later or just keep on creating this backlog of automation, which in reality we never get to. So my strategy is to automate the high priority critical test within the same sprint and include that piece in your definition of done. So like once your test story is tested, you report all the defects, the defects get fixed, it's accepted and you automate it and then you push it out to production. That means you're always up to date with your automation backlog.

So there are any lessons you learned from doing test automation that you iterated on to make better? Like, like how long does it take to run your smoke test? Did you find a bottleneck at first? And how do you get the whole team not to lose, lose confidence in the test? Like if they were failing, were they being flaky? Just tell me a little bit about the real world implementation at that point.

Right. So we wanted to, like I said, refocus on, on a higher deployment frequency. So we wanted to get to a point where this were a particular team where we were doing like monthly deployments and then from monthly we did automate a whole lot and we wanted to get to a point where we were doing weekly deployments and then now we're doing daily deployments of course. But so we were looking into this big test suite and we were having a whole lot of flaky tests. So we wanted this release to be a non event, we wanted to be able to release on demand. So we tried breaking down that problem into different pieces where we looked at the test execution time, we looked at how many tests are failing, we looked at how the test were organized and then how the test results were being reported. So basically the test execution, it was taking us about five to six hours to run run the test. It was about 3,4000 test cases and it was taking like five to six hours for regression testing. So we started collaborating with our DevOps partners and we scaled our infrastructure. I think initially we were running in EC2 instances and then we switched to Docker containers and we ran a whole bunch of parallel tests and we were able to get it down to under 30 minutes of parallel execution. At the same time we were getting into a lots of like false negatives. And for a 3,000 or 4,000 test suite, even if you get like 5 or 10% of tests get failed, it's a lot for a person to sort of review and validate the test results. We basically looked into options like how do we get rid of these flaky test results. So, so we implemented a bunch of things. I remember we did retry analyzer, where we would just look into the failed test, we try to execute them again until they pass. And if they're really a failure, then they would always fail. And then we also worked with our team to identify what tests there are which have open defects. And do we really need, if the defects have not been fixed in forever, do we really need to continue running these tests? So we sort of cleaned that up a little bit, we prioritized our tests, we worked with the developers to sort of, of. So if say a product comes in different languages, do we need to run everything in every language? What, what is the best combination we can get out of this whole big test suite where we're not repeating the test cases? And then the other big problem was how do we report the test automation results? Because we wanted everything to be sort of integrated and be built into the CICD pipeline. So we looked at many options and since we're using JIRA for all our test cases, for our project management of using Zephyr, so looked into Zephyr APIs, I think it's called Zapi. So basically what it does is it integrates the automation test results back into jira. So before that we were using this external dashboard which was created by our team to log and look into our automation results. But Zapi seems like a better, more integrated option for us. So we resolved the test reporting piece from that and then once we collect all of that together, now we could run our test cases in parallel, under 30 minutes and we have our test results back into Jira. And then we create a Jenkins job and we use Slack a lot. So we integrated Slack notifications with the build failures, test run failures, and if they pass or fail. And then that's how we solve that issue of getting to a faster test execution.

I wish people could see the slide you have. You have an example of your CICD pipeline or your flowchart, and it's really cool how you lay out pretty much what you just explained. But it's a really great slide. Hopefully with your permission, I'll be able to put it in the show notes for people to check out. But I thought that was really helpful as well. So how do segment tests, you mentioned if they had defects, do you really need to run all these tests if they are known defects? If it's a language, do you really need to run all the languages? Do you do tagging? Like, how does that work?

Like I said, Joe, we use JIRA for test case management and Zephyr. So I think the logistics were pretty simple. We built different test suites and we sort of, we went through this manual process of like revisiting and organizing our tests again. But I think the important part was to, to have that discussion between the stakeholders, like, is this important enough? Do we really need this test? Is this. So we actually got stuck with certain tests which we couldn't run in an automated fashion. And then when we talk to the developers, to the product owners, and it came down to like, these tests are not really that important. We don't really want to run them with every small release. We could run them with on a cadence, like a weekly cadence or bi weekly cadence. Another thing that I think we should do is look at the usage of your applications. Like, if, if you can get some data out of, like, which browsers are being used, actually being used by the customers and what percentage of your customers are using what percentage of browsers. I think, I think those are good insights to consider on how you want to run your test, which browsers you want to run your test. So we looked at all of that and we sort of came to a conclusion like, okay, these tests are. These are the bunch of tests we want to run on a weekly basis. These are the tests we need to absolutely run with every release. So we didn't use tagging or labeling, but we did customize JIRA a lot to sort of have these different test suites. And like I said, our automation results are going back into jira. So we sort of of created these different categories of tests and organized them.

Nice. I'm just curious to know how you handle this as well. You work for a large enterprise, a lot of teams. Over time your tests get more and more. Do you ever do any sort of pruning or deleting? Do you have any guidelines for doing that? Or do you just keep the test going? How do you know over time from two years ago that you wrote a test? Is it still valid or not?

I think it's very important to continuously review the test cases and what you're testing because. Because definitely I think you have to be proactive enough and, you know, just have these reviews scheduled like on a quarterly basis or something, because over time tests do get invalid. And what happens is we know that, oh, these are known issues, these are known failures, but we don't find enough time to actually go in and clean that up. So I think it's, it should be like, as a good practice, every team should keep looking into what's still valid for a particular product.

Great. And so I think you've been at this company for a while, so I'm just curious to know what it was like before this transformation and what it was like after. Like, do you have anything tangible that showed you that, okay, we made this big change, everyone's involved now in the quality engineering type aspect. So what did that get you?

So like I said, Joe, pushing for the engineering efficiency. So basically it's like getting to market faster and delivering value to our customers in a faster way. We look very closely at lead time and we want to reduce the lead time as much as possible. And oftentimes like more than like infrastructure change. It's more like a mentality shift. You know, where you get into this mentality of shipping small batch size releases. Your deliveries become very low risk because if a lot of commits, a lot of changes are bundled into one big release, it is risky. The regression takes a really long time if it's not all automated. So it's peace of mind for everybody, honestly. Because if you're able to release low risk changes more frequently, then the customers are happy, the stakeholders are happy because they don't have to wait for a month or a week to get their features out in the market. We can just release and deploy as they're ready.

I'm just curious to know, in your organization as well, how do deployments work? Do your QEs get involved with deployment? Like so, like how do QEs, I guess get involved in deployment is my question.

So we want qes to be a part of release process. We don't want quality engineers to be the people who just find defects and really to get into. We really want QE to be wearing multiple hats and become really a full stack tester. So we work with, in collaboration with our DevOps partners, with developers and we were pushing for the CI CD pipeline where the deployments become simple enough so that everybody can do it. We even had a theme last year which said everyone deploys and we literally got everybody on the squad to deploy. Even like the product owners, the delivery managers, everybody deployed the release to production. But again, it has to be simple enough. It has to be one click release to production and everybody has to be involved in the releases. So yes, we want quality engineering to be involved in releases.

Awesome. Sounds really cool what you've done with your organization with this transformation. So I guess before we go, is there one last actual piece of advice you can give to someone to help them with their QE efforts and what's the best way to find or contact you.

Sure you can find me on LinkedIn. My name is Lagan Khare and the one piece of advice is like collaboration I think is the key for all the quality engineering. People like collaboration and relationship building because if the different disciplines are working together, collaborating like development, the product owner, the DevOps, if you're all working together, we can definitely gain efficiencies and make this whole delivery process much faster and less risky. So that's one piece of advice for me.

Collaborate with your counterparts thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a347 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. And if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.