Cypress and QA Strategy with Marie Drake

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we'll be talking with Marie Drake all about Cypress and QA strategy. Marie is the principal Test Automation Engineer at News UK within the Product Platforms team where she is responsible for setting up the overall QA strategy. She's also a tech blogger@mariedrake.com and most recently is a Cypress Ambassador. She really knows our stuff. Really excited to have her on the show today. You don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial, visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

Hey Marie, welcome to the Guild.

Hey Joe. I'm happy to be here.

Awesome. Great to finally have you on the show. I met you at an online event a few months ago for applitools and was really inspired by what you're doing at your company and what you're learning on Cypress and automation. Before we get into though, Marie, is anything I missed in your bio that you want the Guild to know more about?

Yeah. So most recently as well I became the co organizer of the Cypress UK Community Meetup Group together with my colleague Gary Dar. So our aim really is to create a close community within the uk, you know for Cypress users. But however nowadays since everything is now remote it's really good because we also get people like from all over the world join our community and most recently we just hit the 500 mark in terms of our members so we've only been doing this for the past three months so it's been really really good so far. We have done a couple of events so far and we have the following event coming in this August where one of our speaker will talk about Cypress and how use it to leverage doing component testing. So far it's been really well received. We also have a Slack channel that we created for our members so that they can ask the follow up questions after the meetup. And yep, so far it's just been really good so it's given me the opportunity to you know to try and organize meetups as well and you know find speakers who are interested to talk about their journey about Cypress. So Yep, it's been really good.

Very cool. So I definitely will include those links in the show notes for this episode so folks can check it out for sure. So talking about, you know, really expanding the reach about Cyprus, you also, as I mentioned, is a Cypress ambassador. So is this one of the things a Cypress ambassador does is put on meetups and do education type things like this?

Yeah, I think mainly we just want to educate everyone about the tool as well. So one of our goal is to try to spread the word. If we have any sort of new features that we want to showcase, it can be either via a video course, like a tutorial or a blog post. And it's just really to help out other people as well if they're stuck with any of their Cypress queries.

Nice. So since running a meetup, is there any topic you hear coming up over and over again that maybe people don't understand about Cypress or maybe confused about it or don't know there's a feature that it can help them with their testing?

Yeah, I think from the previous meetups, we always get a lot of questions on, you know, how is this different from Selenium? Is this going to be the replacement, you know, for Selenium? And we just go on and say that Cypress is a completely different tool, its architecture is completely different. And I know that Gleb also mentioned it on one of his podcasts where he talked about Cypress. So I think it's just to educate the users that it's not going to be a replacement for Selenium. These two tools can work alongside together. And I guess with all the different testing tools out there, you just have to do a lot of analysis, a lot of like, POC and to check which of the tools will serve your testing requirements. And we always just say that it's not going to replace Selenium, but it's an alternative if you want to use like a different tool and it's just there. And it's not really going to replace the tool like Selenium because as I mentioned, it's two completely different tools and it both has its advantages and disadvantages. So you just have to balance out which one would meet all your testing requirements.

Absolutely. Well said. You're responsible for setting up the QA strategy at News uk. Is Cypress a tool? There's a lot of tools out there I think you were going through, like how you can use the same technology across all the different verticals. So people reinventing things. Is Cypress a tool you use as one of the standards that every team uses across the different silos at News uk.

Yeah, definitely. So most recently we actually have like an official RFC at News. So it's just a document outlining all the different testing tools that we've recommended and we actually put both Cypress and Selenium in that document. And basically if because most of the projects at news they're utilizing JavaScript testing frameworks such as React, so in that perspective it makes sense to try and use Cypress, especially because we also want to share the responsibility of writing the test. So it's not just the tester's responsibility nowadays, it's now seen as everyone's responsibility. And what I found is if you give your team or your developers a tool you know, that they would actually use on their day to day basis, then we get more buy in. So the reason we put also Selenium as part of the RFC is because other projects, they have different testing requirements, they also have to do a lot of cross browser testing. So in that perspective it makes sense to use Selenium. But if you're just using or if you're user base is mostly on one browser and you're interested more in sort of getting everyone to help out with testing and test automation so you can focus on other testing activities, then we've started recommending Cypress to the different teams here at News.

How's that working? Are developers embracing this tool and any tips you have to get developers to really get more involved in testing?

Yep. So in one of the projects that I'm part of, so in product platforms, because it's a greenfield project, developers write the test for us, but in terms of setting up the frameworks, doing the PoCs at the beginning we help out a lot so that at least we lay out the different options that we can use. But nowadays I haven't even written like a Cypress test for that specific team because they're just doing it all. And it's really good because that means that I can focus on helping out other teams where I'm most needed and I don't have to worry about. Okay, have we actually covered a test for this feature? Because we add it as part of our acceptance criteria that before we sign off this ticket there should be some tests that are written, whether it's on the unit level or on the component level or on the end to end level. So in that case it's actually been working great for us so far. I know that with other teams where there's still a dedicated test automation engineer still sort of writing the test, but I think slowly we're getting into that culture of everyone owning the test. And I think it takes a lot of time in terms of doing all these changes, but I think we're going on the right path.

Cool. So how do you verify or validate that developers are creating the right tests at the right level? I know if everyone starts contributing and they start checking code in, is there like a team that actually looks and verifies that there are coverage for the important tests that you need to have your customers?

Yeah, so we do this as part of our code review process. So one of the key things that I've actually sort of introduced within the new teams that I'm working on is to create a pull request guideline that also covers the types of tests that they need to run. So before we pick up the ticket, some teams, they have like a tech analysis session where they talk about, you know, the different technical steps that they have to do. And this actually includes what levels of tests are needed. Once we've written down all the details on that test, it's just kind of like a checklist on every code review. For example, if you've written a test for this, make sure that that's also included on the pull request. And if it's actually needed to write, let's say, an accessibility test for a feature, then that also gets loaded as part of the tech analysis session and also verified on the code review process.

Nice. So in one of your presentations, you mentioned a tool, it's called Storybook, that I thought helped your teams reuse things. So I guess. What is Storybook?

Yep. So it's just really a tool to sort of visualize all the different components that you're using. So we use it heavily within product platforms because we're building a design system, you know, which different title teams going forward are going to use. So the idea is the Storybook would just visually show all the different components that we have. And on each component, it can have one or more scenarios. When I mean scenarios, it's like a different variant. So if we talk about a button, one scenario for a button could be disabled. It could have an animated sort of loading button, all those sort of things. So it's just a place for us to visualize and see all our components in general.

So at the beginning, you mentioned component testing for the folks that don't know what is component testing.

So I guess it would depend on, you know, what you're trying to test. In our context, component testing means testing a single component. So it's just testing, you know, for Example, if we're delivering a button component or a text component, we make sure that we have a test for those individual components. So the types of tests that we do, it could be functional testing on the component level, where if a component is quite sort of more complex, so for example, we have a component for our radio player which is a combination of one or more components, then you can add functional testing on that level where you can test the toggle of the play and pause button, the fast forward or the rewind, all those different things. And then at the same time, you can also do accessibility testing on each of the components, just to make sure that each component adheres to an accessibility standard. So rather than waiting at the end until after all the different components have been sort of integrated together and then write the accessibility test, you can actually test those components in isolation and then check for different accessibility issues.

At the same time. You can also do things like visual testing per component level. If you just want to, you know, catch some visual issues on a component basis, then we're doing that too. So it pretty much depends on what sort of project is. But I guess on a high level, component testing is just testing a component or a block in isolation without really integrating with other services.

Nice. I guess. Is the benefit that it's faster? Yes, because you know you're making a change to that one thing. You can map issues to that exact thing.

Yeah, definitely. And we also know. So if there is particular issues, we know which specific component is having that issue. So I guess it's more useful as well in terms of debugging it quickly.

Well, so you mentioned two different types of testing as well, Accessibility testing and visual testing. So accessibility testing, is that a newer approach that you've added to your test plan, or is that something you've been concerned about from the beginning? I guess being a news organization, does that play into why accessibility is so important?

Yeah. So I've actually thought about doing or introducing accessibility testing in the past, even if I wasn't part of this team yet. But it was because the project that I was on previously, it was already a live product. It was quite difficult to start prioritizing accessibility testing. So when I got the opportunity to join a greenfield project, it's completely brand new. I decided to introduce accessibility testing at the start and get everyone sort of like ideas on how we want to do this the right way. And I think it helps as well that we're trying to deliver a design system where our accessibility standard is a must, because different title teams will use our Components. So in that perspective, it just made sense to integrate it as early as possible.

Cool. And also, you talked about visual validation testing, so how does that come into play? What does that help you with? What types of testing to use visual validation for?

So we do two types of visual testing within the current project that I'm on. So first is on the component level, and within this component level, we utilize a tool called applitools. And what's great about it is it actually hooks up to Storybook automatically, so we don't even have to write additional tests as part of it. All we need to do is write the configuration file, decide which sort of browsers or mobile viewports we want to run our components as. And if one of our developers created a new component, they don't have to do anything. Applitools will just pick up the new component automatically and then we'll just do all the baseline and the image comparison for us. So we do that on the component level as well, because we want to make sure that if a title team use our individual components, then rather than them finding the issue, we want to find it as early as we can before getting it deployed to our end users. And then on the other hand, we also do a full page visual snapshot. So apart from the different components, we also maintain a documentation site which serves as a living documentation that the different title teams can use. So we have to make sure that on the documentation side, every time you make some new changes, especially on the CSS or on the design side, that all the changes are all expected and we're not introducing new visual issues. So we're also utilizing the same tool which is applitools, on that type of test in presentation.

You brought this up. You started with some open source solutions, so could we talk about maybe some open source solutions? I think a lot of people think that it's just vendor based solutions and then maybe what the differences are between the two.

Yeah, definitely. So we actually experimented with a lot of open source visual testing tools out there. So we started out using Backstop js, which is a very popular visual testing framework based on JavaScript. Initially, backstop JS works well for us especially, you know, because we just started to introduce visual testing. However, the problem that we found was it was only running on one browser and we needed something that can run in other browsers as well. And similarly at the same time. This is a pixel by pixel comparison tool. So if there's a one pixel change or if we have dynamic data, then we always get that Sort of failure in our build. We have another team at News UK who actually created another visual testing tool called EyeSpy. And this is to address the limitation of backstop JS, for example, because I Spy hooks up into your Selenium grid so you can run your tests on different browsers such as Chrome or Firefox. However, we were utilizing the official Selenium Docker images, and during that time we found out that they didn't really have images for Safari or Internet Explorer. So we were still a bit limited. And again, this is a pixel by pixel comparison tool. So we would end up a lot of maintenance, you know, just trying to make sure that our test is very stable, but we still end up, you know, having all this, like, flaky tests. And that's why we opted with using applitools because we wanted to try something different. And when I was looking at applitools before, they weren't using pixel by pixel comparison. So they're using different machine learning algorithms and they have a lot of comparison levels that you can use. And it also allowed us to run the tests on different browsers, which was a massive win for us because we need to make sure that we can also handle visual differences on other browsers, you know, such as Safari or Internet Explorer. So I guess if you're just starting out with visual testing, I still recommend looking at the open source tools because at least you have some coverage in terms of visual testing just by using open source tools. If there's sort of like a tip that I would like to say is make sure that you use Docker for generating the baseline and the test images. And the reason I said that is because in the past we always have a lot of this pixel by pixel differences. And it was because we were generating the baseline images on our local machine, and then the test images are getting generated on our CI server, and both our local machine and the CI server doesn't have the same configuration. So if anyone's experiencing this, I highly recommend using Docker to try to make sure that you're running your tests on the same configuration, on the same node version, for example, on the same Cypress version, all those kind of things, and it will help in sort of reducing the pixelated differences. But if you want more in terms of you want to cover more browsers, you want to make sure that you can also compare dynamic data, then you can look at options like applitools. But make sure to obviously talk within your team, because applitools is a commercial tool and it all depends with the budget and everything.

Awesome. Doc is a great tip. I know you also talked about flaky test pipelines. A lot of times people just talk about flaky tests. So any other tips to create a pipeline that isn't as flaky or that isn't flaky?

Yeah, so there are a lot of tips. And I guess in terms of the other experience that I have, the first sort of thing that I look is make sure that this test selectors that you're using are actually really stable. So most of the times we use things like ID or class names or CSS selectors, but if you're using, I guess if you're working on a project where these selectors are auto generated, then it makes sense to introduce a stable or reliable way of identifying these elements. So you can introduce a custom attribute like a data test, ID or data test, for example, and it's just a sort of a small thing that you can add if you have access to your code base. It's so easy to just add the custom attribute, or if not you can just ask, I guess, one of the developers to do it as part of a ticket that they're working on. And it actually really helps because at least we don't have to worry about updating our selectors. If the selectors on our application are also changing because we have this custom attribute, then I think there's other tips as well. You don't have to do everything on the UI level. The problem is doing test automation on the UI level is already hard on its own and there's a lot of complexities and you can actually leverage some ways on. For example, if you have a login service that is a prerequisite for you to access your application, rather than doing the login on the ui, you can try to utilize the API endpoints if you can. And if you don't know how to do it, just ask your team, ask your developers for help. Don't struggle by yourself, because everyone's just going to help you anyway. And it's just really good if you can push back some of these tests on the API level, because they're not only faster, but it will also contribute in a much more stable pipeline because you don't have to worry about logging in the UI and then doing some hard weights just to make sure that the login is successful. And what I've seen is some people will actually put a lot of weights on their tests, and I'm not like 100% against it, but if you can avoid Adding like hard weights, then please do. It's better to look at how testable your app is first because your test automation or the tool that you're using will only be as fast or as reliable if the application that you're testing is also reliable. So it could be an opportunity for you to say, oh, we might have to improve the testability of our app to, you know, first before we can resolve all this different flakiness.

I love the API testing tip. Can you do API testing with Cypress?

Yeah. Yeah you can actually there is a built in method that you can use called side request and this by default will do a get request, but it also handles the other action verbs like pose or delete. And I guess it's actually one of the advantage that I like about using Cypress is you don't have to install additional libraries or additional sort of methods just to do an API test because it's already built in. So you just have to do sign, request, pass in the API endpoint that you would like to query on and then just proceed with your tests.

So what I really love about you is you're not focusing just on ui. Like so far you've talked about API testing, you've talked about visual testing, accessibility testing. So I really love this approach. Component testing. I think also on your blog you talk about performance testing, which blew my mind because not a lot of testers know about performance testing. So do you do any type of performance testing as well?

Yeah, it's actually very recent for me because I've noticed that most of my posts are focused on functional testing requirements and I read this post by Christine Jacobani before where she talked about the test automation wheel and it actually gave me the idea that as testers we shouldn't just try to focus on the functional tests or the API test or all the other types of functional testing that you can do. Most often we tend to forget the non functional side as well like performance or accessibility. So the reason I did that post as well is because in one of our teams we're using Torus, which is an open source tool that was developed by a blazemeter to help us with our performance and load testing. And I wanted to show to the community that it's not as hard. You just have to create a JSON file and then they have all the different sort of properties that you can set, like what's your concurrent users, the ramp up to time the request that you need to test. And I think because prior to that the most popular open source test automation tool in terms of performance is JMeter and it's still very important. But when I started using JMeter, I found it very confusing. So it's quite good that nowadays we get this new tools that are much more easier to understand. So at least if you don't have a lot of, you know, experience in doing test automation in performance testing, you don't really have to worry about it as much because the new tools nowadays, such as Taurus and I've even heard one called K6, these are much more easier to use and much more user friendly and you can definitely ask the team, like the whole team to help you with performance testing. So as a good example, when we wanted to introduce performance testing, we created the ticket ourselves that, okay, we're going to use Taurus for performance testing. So in terms of picking up the tool, I was helping the team, but the work itself, in terms of actually writing the load test, it was actually done by one of our developers. And what I like about it is they didn't have to rely on myself because the new tools nowadays are much more easier to use. They can just read at the documentation, look at all the different examples and then if they need any more specific help, then they can ask. But yeah, so far it's just been good. We're quite lucky that we're at the position within the team at product platforms where everyone is sort of trying to be responsible for the whole quality and for the whole testing and we haven't really got a lot of sort of like setback from other people that, oh, okay, this is your job, like, why am I doing this? Like all those sorts of things. So it's just been a great experience so far.

Okay, Marie, before we go, is there one piece of actual advice you can give to someone to help them with their test automation efforts and what's the best way to find or contact you?

Yep. So I think if I have one advice is don't stop learning. Just because you use or you've already learned one tool doesn't mean that you're already like gonna stop your test automation journey. Because technology nowadays is always changing and especially if you're working in JavaScript, there's just always like some new testing tools that's coming up like every other week or so. And Cypress is just, you know, one of the tools out there. There's a lot of other tools out there that you can also use and just utilize all the different free resources out there. So there's a lot of free courses now. There's a lot of some really good podcasts that you can listen. There's a lot of, you know, really great communities out there that you can reach out to. And I guess if you want to reach out and you want to know more about me, I'm always on Twitter, so my Twitter handle is mcruse Drake. You can also contact me via my blog, there is a contact form in there or you can also connect with me via LinkedIn. It's just Marie Drake. And yeah, I'm just happy to answer any questions that you have or any sort of like Cypress specific questions that you might have as well.

So yeah, thank you Marie for your automation awesomeness. For links have Dinner Valley we covered in this episode. Head on over to Test Guild.

3:16 and while you're there, make sure to click on the Try it for Free Today link under the Exclusive Sponsor section to learn all about Source Labs awesome products and services. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.