Scaling Automation Testing at Startup with Priyanka Halder

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking about Priyanka Halder all about scaling automation testing at a startup. I saw her at an Apolitools event and what she did has blown me away. So I'm really excited to share her experience with you today. Priyanka has over 12 years of experience and quality assurance in various startups like Goodrx, Heal, Home, Me Inc, Truecar and she currently heads the quality engineering team in Goodrx, which I believe is almost, I think it's a multi billion dollar company. So she handles a lot of stuff there and I'm really excited to have her share that. Today she really is focusing on building her career, implementing QA teams to use technologies like visual validation, test stabilization, pipeline, CICD for each code deployment and much more. If you want to know how to succeed with automation, have the lowest amount of failure rate I've ever heard of any company with automation. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Priyanka, welcome to the Guild.

Hey Joe, thanks for having me.

Awesome. Before we get into it, is anything I missed in your bio that you want the Guild to know more about?

I think you gave a fantastic intro, but I just want to add that I'm doing this for almost 14 years. The quality engineering and one interesting fact about me is I have represented USA in a Bug Battle Championship back in 2014 and I came first from USA. So that's Icebreaker. I can say people might be interested.

To know what is a bug championship? How did that work?

You know. Applause it's like a crowdsourced platform so they used to hold every year worldwide. You could submit your bug on any open source tool or even like license tool and I submitted a bug on jira.

When you like entered one of the.

Field it will just break JIRA altogether. And my bug got selected, nominated for top 10 bugs reported and then all over the world people voted for the best three and mine was in top three so it was like third worldwide. But first from us.

Very cool. So Priyanka, I want to dive into what you've done at GoodRx and you've created a really cool framework and you've had a lot of great success with it. So I guess before we dive into that, give us a little background of what GoodRx was maybe before you got there and what challenges you saw that had to be solved with say an automation framework that could help with testing.

Absolutely.

So CudRx is one of the companies who sustained their product without a quality engineering team for almost six years before I joined back in 2018 October. So how they managed the development team was pretty small. They had their own automation framework which all the developers were writing together in Bok Choy Python and they had that in their CICD pipeline. But it wasn't working at all. Like every time they will run it it gonna run into a CSS problem because like lot of people are writing it so they will not know like how they're breaking the other tests. Right. So it used to be a nightmare, especially with that automation framework they had. And because they didn't have any quality engineering team. So there was lack of processes like how a project will go with better quality to production. There were a lot of challenges of pushing a very big project with minimum number of bugs in production. So that's how it looked like. And they had only one QA engineer before me who was just supporting for four months before I joined. So when I joined it was interesting because I got the opportunity to get a blank slate and then write my own destiny on it, like write my own framework, get my own people. So that was a very awesome experience for me to start with.

Nice. So it is a startup and when I hear what startups is, the trend is not having a QA team and having the developers do everything. But it sounds like that's what happened at GoodRx. But you kind of came into the picture. So does your developers still create automation or do you have a separate team? How does that work?

Back in 2018 we started to change the process a bit. So when I came into the picture first I understood where the pain point and what kind of quality they were looking for. So the main point came out was they wanted to not break our bed butter features like searching for coupons, searching for pharmacies, getting a coupon. So we figured out what are the critical path is and then we automated. So actually I hired senior level estates to make that automation happen in first three months. And whatever you see, I talk about a lot Credit has to go to that person called Vahan Milken. He is the brain behind a lot of the cool gigs that we have done together. So though I get the pleasure to talk about it like his contributions are endless to this. So again like when I hired that person and when we like sat together and decide what we going to do. So the first requirement was we will be creating a framework and putting it in our CI CD pipeline. So all the developers at that time it was maybe only 30 developers were there in Goodarex back in 2018 and today there is 180. So our first goal was to give them the assurance that okay they can push anytime if it is front end or backend changes and with help of our critical path automation testing. So that's where we started. But like if you see today we are now changing the whole game and now we opting for developer to write more. So it depends where your company is. So if it is a small startup it's easy to have a service based team and then spread it across. But now GoodRx is growing at a very much larger scale where I do not have the privilege to have a very big sdate service SDA team. So now they are embedded and we expect everybody to write tests.

Very cool. So what were some high level principles you used that you wanted to get into a framework? Are there anything you always require from a framework you should have before you started developing it?

Absolutely. So like when we started developing it, my first mantra is learn. Learn from failure and learn like what exactly the problem is. I never join a company and jump to a solution like what I think is good, but I try to customize it what that company is need at that particular time. So the first one like first few weeks we just learned like what are the challenges, what are the processes, why the old framework did not work right? Do we need to use some kind of strategy like visual validation? Because this is the first company we are using it and Goodarex is very heavy on content. We show drug contents like drug classes and it's very important that they are right. Like you can have a lot of different kind of social media which are heavy for content. But I don't think people will be hold accountable for if something is wrong in that content. Right. But because the industry GoodRx is so one of the core requirement came up that whatever we display it has to be absolutely perfect and not distorted and like look and feel has to be really good. So like when we did not have the automation framework it used to take like One manual quality engineer to do it. Almost six hours to go through the pages and make sure and still bugs will make it to production. Right from there we understood that okay, we have to reduce the time we have to have visual validation. So like that as we started learning, we started creating like adapting those in our framework. But one of the second core thing we do is we always look for the industry standard. And Source Lab has been very helpful with that. They publish industry standard every year. They did it in 2019, they did it recently in 2010 where they show us like who are the top players in industry and how their automation framework looks like. So I and my team learns from it immensely and we try to make sure we are following all the industry standard that they are telling us that that's how the company is going. So we are pretty up to date like where the industry is. And based on that we create our framework as well.

So look, a little side note here. I believe like I said, I've seen that you do a presentation around this earlier with visual validation. I think a lot of people fail to realize how powerful it can be. I think you said you found a logo issue with it you wouldn't be able to find even if you were trying to. So could you just explain a little bit how visual validation helped in that particular user case?

Absolutely. So visual validation is another step ahead of functional validation. We can write lot of code in Selenium and say how what the size of the logo can be. Just think about a logo, maybe goodrx.com logo but it still will not be able to say that yes, it is showing it right on a particular screen or a particular browser. That's why like applitools tells it very nicely. It's like adding I to your functional test. So that's where visual test comes from. And in goodartx we have found immense number of bugs which is very difficult to find with bare eyes. One example was GoodRx supports Kroger SC.com that is like Kroger's biggest pharmacy market. And recently they have changed their logo. So when we were having one of the production releases, our visual validation found that the logo suddenly changed to the old logo. So it was not matching with the baseline. So if you think about a person who is doing that validation manually, it's almost impossible to know that okay, a logo has changed like our human eye does not catch. Our brain does not work like that because we are so used to see logo a certain change does not. It's not easy to catch in with our bare eye. But like with visual validation it's super simple. You have a Basel. You just tell okay, I'm expecting this. And the moment that even moves one pixel it tells you that okay, there is something wrong with it. And you can go to their dashboard and say either it's a bug or it's not a bug and that way you can catch it really easily. So that was one of the use cases. But we have like number of bugs that it has found one day like we were not able to select, like one of our configuration was not rendering properly on ie. Like we find oodles of bug with using visual validation as well. So there are quite a few examples.

Cool. So I'll have the slide in the show notes but you have what you call your next gen SWOT framework and so you use Selenium Appli tools on the top end and then next is you do use bdd. So I assume usually you mentioned Python. So using Python, which BDD framework are you using for Python and how's that used within your team?

Sure. So we use Behave, that's like the similar of Cucumber for Python. Right. Behave framework. We also have custom code to run parallel test with Behave. So how we use today is that whenever our product person come up with requirement, some of the product guys actually started writing in BDD which is the three amigos rule where the developer tester and the product work from the requirement phase in bdd. So some of the product people give us the happy path in BDD or maybe they give us in a normal requirement form and the quality engineer goes and then write the BDD from there. Once it's written, it's shared with our developer and developer and the product manager gives a thumbs up saying that yes, this is what they were expecting. And then either the quality engineer or the SD goes and implement them. From the BDD onwards we have a ground rule. So if any functionality that go to production, if it is priority one or zero like critical path, we automated before it makes it to production. And those automations are already in our pipeline. But if it is a P2 means like if it is broken we can sustain it for a few days. Those automations are written in a catch up fashion later on after the functionality is published already in production. Some of the benefits that we see saw on BDD is like it's super easy for even developer to know what is breaking. So if they're pushing something to production and suddenly one of the tests fail, they can just go look at our failures and Just repeat those tests by hand manually and see if it is an actual failure or if it is a flaky test. Because our automation framework is so stable and we rarely see any flaky tests like it's easy for them just to check the fail test case and see where the bug is happening. So we see events like positive feedback from Even developer from GoodRx like most of the teams, I will not say 100% but most of the teams who likes to work in this fashion.

Another unique thing about your framework here is if you talk to people, BDD gets a bad rap and they'll show you examples of poorly written bdd. It's hard to maintain and no one contributes, but it sounds like you've kind of solved that somehow. How did you get people to embrace BDD and how do you maintain the quality of them?

I would say it depends on team to team Joe, because if I give example, suppose we have a team who just work on SEO features, site optimization. For them BDD might not be the best option. So we don't force them to use BDD in that case. But if you look@goodrx.com which has lot of functionalities, lot of visual validation, it becomes very easier for them to get the help from quality engineers and estates. Also like the fashion we write it like when developer developing it, our QE engineers already written the test so it gives us the use like leverage. Our quality engineering, our estate, our product and our development team together also with reporting. So that's how we sold it. And like my higher ups, like directors of engineering, they were already sold on this. So I would say it was a positive point. When your higher ups are sold on some concept already it's become easier to push it through. But again we started with one team, we showed the benefit and then other teams started accepting it. But at the same time we understand it might not be the best fit for all kind of features, all kind of teams or all kind of companies.

Great. So you mentioned you have to test a lot of different devices on different kind of combinations. So you're using Selenium, are you also using Appium? And if so, are they all in the same story your bdds and then it knows what to run against.

Yes, we definitely use apm because our mobile web test are ran in actual mobile devices. We do not use simulator emulators today. Also our Android and iOS natives apps are written using native app tests are written using APM. So we do use APMs. Yes, one of the core functionalities, our awesome Framework is we do not have separate repos for different kind of different support for Web, mobile, web, iOS and Android applications. So what we do is we use tagging. So if one test is valid for web, mobile, web, iOS and Android, you will just see different kinds of tag added to that. And same step definitely like step definition will be created to support different kind of platforms. But the test will be only one test because like if you use a pharmacy search, like a prescription search on goodarex.com or mobile goodarex.com or mobile app, the functionality is same. It's just the platform that gets different and maybe the look and feel gets different. So we just tweak that in our step implementation. But the file like BDD remains same. So it's only one repo mono repo and it's supporting all the different platforms.

Nice. All right, so you have your technology selected, you have folks writing test and bdd. They're going against web and mobile and using visual validation testing. So you have that all in place and then it gets what someone submits a code merge to it and then tests run. How does the next step happen?

Absolutely.

So we use Pinnacle pipeline. So every time a developer pushes a code to master branch, it gets deployed to our staging environment. And there we first run all our regression tests using that pipeline with six cross browser and two mobile web browser for goodalics.com for iOS and Android app. Every time a build is made, we upload it using Bitrise and again all the functional core tests are run against the build. One quick thing I want to mention here is we do some a very unique thing that is called device fuzzing and that's also very unique to our framework. So what we do is because lot of our users are on various kind of different devices every month we make sure with GA like Google Analytics that the top browsers and mobile devices we are using are the devices and browsers that our users are using. And using that device fuzzing code, we make sure that runtime we are selecting top 6 browser based on the GA. If suppose one test fail on iOS X like iPhone X, we make sure that when we are retrying three times, we are selecting the exact same devices when we are rerunning the test. So that is something very unique that my team has implemented.

Awesome. So you did give a meet up on this. You showed something to do with cookies. Is that something else besides what you just mentioned?

Yes, this is absolutely different code. So this cookie is one of the different codes where we utilize it to Say that this is our staging environment. Let us go through the cdn. Right. Which stops any outside request to come into our staging environment. So we don't use any local testing for our cloud provider like either. You could use local testing and you can whitelist the tunnel, but it makes your test slow. What we do is we use a cookie and we tell the content delivery network proxy that hey, this is a test, let it go and let it run on our staging environment.

Nice. All right, so you run the test they run in the cloud, they run, gets all the things and then they report out. So I think now you do something with AI, talk a little bit about reporting and how to triage. What's the next step?

Sure.

We have two different kind of reporting dashboard. The first one we use it called ReportPortal IO again a very new.

Machine learning and AI driven reporting portal. Absolutely love it. It gives us a way to see that what are the different builds running at different time. It also gives tells us with it machine learning that if production bug or it is environment related bugs. So absolutely, the more number of time you run and more number of bugs you find, it automatically learns that what kind of bugs are automation is reporting. Right. The second thing is do is debugging. So using report portal at one place you could actually see all the steps in bdd. Also bottom of it you can see exactly where it is breaking and why it is breaking. So again for developers it frustrates them to go to Jenkins and see Jenkins log or go to browser stack or see like find and see browser stack. Because we are writing, we are running 16,000 tests a day, right. So it becomes very hectic for them to search, find and do it in different place. So that's why we have consolidated everything and we show it in one place inside our report portal dashboard. So that's working. And it's connected with our GitHub so they can log in using their GitHub login SSO and they can see it anytime. The other dashboard, we use it for our internal stabilization. That is like we are constantly monitoring which tests are taking more than two minutes, which browsers are the slowest browser and why, what's the case on it, like what time during the day we are running most number of tests and can we optimize the way we are running it. So that's our datadog dashboard and that's for our QA team only who are constantly monitoring the health of the test. So both helps immensely to carry forward this kind of large scale automation at Goodrx Nice.

There's so much to cover. So now I just want to go to the wow. The wow factor here. So you have your framework, you have the team bought in, you're running them. How many times do you run it? Give us some metrics. How stable are your tests, what's your failure rate, things like that.

Yeah. So we almost ran 16,000 times a day. In last six months we have ran 3.1 million tests in cloud. And then every time a developer pushes to a test environment or a staging environment or production, like almost 650 test cases ran cross browser within six minutes. So that's some of the stats. So it's huge number. And when it comes to error rate, as you told, we have 0.0, I think 1 2%. I don't have the slides in front of my eyes, but it's very, very low. And I think 2% test failed. But most of them are valid failures. They are not flaky tests.

All right, so this is going to blow some people's minds. A lot of times people blame a tool like Selenium. Selenium is frigging awesome. It's usually the way people design tests that create flakiness. So what makes your framework so stable? Like what did you do with Selenium? Did you have better weights, locators or how did you get your failure rate so low?

We do a number of things to make our Selenium test faster. Some of them are is use of tagging. So we use tagging in a very clever way. We use it as a utility tag, as a project tag and as a priority tag. So that saves some time. Like how we are using, for example, if we have to enable or disable visual test, it's just use of a tag and we could do it within few seconds. Right. So if we see that our tests are taking longer because maybe applitools is having some issues, it's running into problem. We can disable the visual validation test like within seconds using utility tags. So the other very cool thing that our SD did is lazy loading of elements. So instead of creating each element at a time, as we are clicking through the pages, what we do is we land on the page, we create, initialize all of them and then we lazy load whenever we are like interacting with that particular element. This has drastically made our test pretty fast as well. And the third thing we do is we wrap lot of our objects and we club together Selenium commands. Right? So instead of having too many Selenium commands, calling, getting called by one of the test we do is we wrap it under one of the object and we use use it. So that optimizes our test a lot. Like these are some of the tidbits but there are series of things like cool techniques this we use like for example for mobile automation. Right. You know that when you first initialize actual mobile devices in cloud, it takes maybe 15 to 20 seconds to enable that. To optimize our APM test, what we do is we utilize debug capabilities. So we have told our developer to give us a way to land on a specific screen. Instead of going through introduction every time or instead of registering every time, we make use of APIs whenever it's needed. So instead of going through the all pharmacies we might be able to see beforehand what pharmacies are thrown by our API and what particular pharmacy we are targeting. So there are series of small things that my team does to make sure it's that level of stable and it's like everyday work and that's this is the fruit of like one and a half years of making this automation framework.

So Priyanka, I also know you're using AI. Has it actually helped you to reduce time? Can you just give us a little info how AI has actually helped your framework as well?

Absolutely, Joe. So as mentioned before like previously with just manual testing it used to take us a regression almost six hours as we enabled like visual validation and report portal reporting. Our whole pipeline runs within six minutes, 650 tests with thousands of visual validation checks it has optimized almost 98% of our time. So now we could go to production within like 10 minutes, 10 to 15 minutes if you take all the steps that it does in Spinnaker pipeline. So it has helped us immensely by integrating a visual validation tools and a reporting portal which embraces machine learning and AI.

Nice. So it sounds like just the time it saves from triaging using AI has been helpful Visual validation. Are you doing anything with when you check in code? I know there's a lot of tools like C Light. There's a bunch of them when you check encoded it knows okay they changed code A, B and C so therefore I only need to run Test 1, 5 and 6.

We are not doing it currently but I do plan to check out AppShorify. Like James is my friend, I met him in lot of conferences and I do want to try that out in future. And C Light also you mentioned in the meetup definitely on our page to check out like if that can also improve more our framework.

Yeah we had James on the show. He has a really great solution there as well. So that's another great option also. Okay Priyanka, before we go is one piece of actual advice you can give to someone to help them with their automation testing efforts and what's the best way to find or contact you?

Sure.

So I am pretty active on Twitter Pre Tech Mom PreTechMom is my handle and they could find me. They're pretty active there. So reach out to me on Twitter or even add me on LinkedIn. I'm pretty active there. So the last piece of advice I will give to everyone regarding automation is so no one tool is perfect for every solution and no one tool is like slow or fast or does one thing good or one thing bad. It's always your solutions, how you utilize that tool and how you utilize towards your product. So again, understand the product you are testing, understand the goal of that product, what they want, where exactly they want, the quality, and based on that, choose your tooling based on that, utilize all these recipes that I shared and make your automation super solid.

Thank you Priyanka for your automation awesomeness. Links to Entertainment Value we covered in this episode. Head on over to testguild.com a318 and while you're there, make sure to click on the Try it for Free Today link under the exclusive Sponsor section to learn all about Sauce Lab's awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.