Performance Testing using Cypress.io with Marie Drake

Welcome to the Test Guild Performance and Site Reliability Podcast where we all get together to learn more about performance testing with your host, Joe Calantonio.

Hey it's Joe. And welcome to another episode of the Test Guild Performance and Site Reliability Podcast. Today we'll be speaking with Marie Drake all about performance testing using Cypress IO. If you don't know, Marie is a Quality Engineering Manager, a Cypress Ambassador and a Tech blogger@mariedrake.com. she has lots of experience around automation, but especially around Cypress. So really excited to share with you a session Marie did for us last year at last year's Automation Guild conference during our DevOps days of the event. If you don't know, Automation Guild is a five day event that happens every February and the first three days are dedicated to functional automation and then the last two days are dedicated to all the other types of automation testing I think is important like performance testing, security testing, DevOps, accessibility testing, chaos engineering, all those other types of things and types of testing that should definitely be part of any testers toolkit. So if you don't know, I'm just about to open up registration for the 6th annual Automation Guild which is an online event occurring February 7th to the 11th, so I'm going to open that up very soon. If you haven't already, make sure to secure your early bird discount now by joining the wait list to get alerted as soon as I open the cart, which I plan on doing in the next few. So if you really want to accelerate your automation efforts which include performance testing in the new year, this is the must attend event. So head on over to automationguild.com right now and grab that Early Bird special. And I understand this might be hard to follow along because I actually grabbed this audio from the video session that Marie gave at last year's Automation Guild. I'm going to try to embed this in the show notes later on and I'm also experimenting with audio only tutorials. Some people have told me that a lot of people are getting value from this type of content, so I thought I'd do a quick experiment and do an episode on a actual audio tutorial on how to follow along to create Cypress to be able to create performance tests for you with Marie. So let me know your thoughts. Drop me a line@joetestgill.com if you want to hear more of this type of content.

This episode is brought to you by SmartBear. Listen, load testing is tough. Investing in the right tools to automate tests, identify bottlenecks and resolve issues quickly could save your organization time and money. SmartBear offers a suite of performance tools like Load Ninja, which is a SaaS UI load testing tool, and Loaduipro, an API load testing tool to help teams get full visibility into UI and API performance so you can release and recover faster than ever. Give it a shot. It's free and easy to try. Head on over to smartbear.com solutions forward/performancetesting to learn more.

Hi everyone, my name is Marie Drake. I'm a software tester and a Cypress Ambassador. I'm also a tech blogger@mariedrake.com and also the co organizer of the Cypress UK community meetup group. I'm also a course instructor at Test Automation University with a course focused on accessibility testing as well as at Ministry of Testing with an upcoming course about Cypress. I would like to welcome you all to my first Automation Guild session which is about performance testing using Cypress and Google Lighthouse. So in today's session I'll give you an overview as to why you should care about your application's performance. Next I'll talk about performance testing and how you can use Google Lighthouse, a free and open source tool that you can use to test the performance of your websites. And finally, I'll jump straight ahead to the hands on session of setting up Cypress Audit, a plugin that you can use to extend Cypress so you can easily run web performance tests and track your Google Lighthouse score. So why you should care for performance well, imagine you're browsing a website to order a gift for someone. You're scrolling down and finally you see the item that you want to buy. You click the image thumbnail but the page is not responding as quickly as you want and after seconds of waiting you decide it's better to open another tab and go to a competitor's website. We use technology on a daily basis and have been accustomed to instant gratification. So if we see that a page has poor speed, we are quick to go to a competitor's website rather than wait. It's a known fact that slow websites frustrate us. Slow websites not only frustrate users, but businesses can lose considerable amounts of money if their websites cannot keep up with the customers expectations.

Now that you know the why, let's talk about what performance testing is. Performance testing is typically divided into two backend performance testing and front end or web page performance testing. With backend performance testing, the aim is to ensure that when multiple requests are sent from different users simultaneously, you need to test that the servers can handle the load accordingly. On the other hand, when we speak about front end performance testing. The aim is to test how fast our users can see the web responses instantly. The way different browsers process data from servers has an impact on performance. For example, JavaScript makes website interactive and fully functional, but it can also contribute as to why your website is slow, especially if it's not optimized. There are a lot of tools nowadays to choose from to help you test the performance of your website. Here I'll focus on one of the most popular tools out there for web page performance, which is Google Lighthouse. Google Lighthouse is an open source tool which lets you audit the performance of your website. It also audits other areas such as accessibility and search engine optimization. To get started, there are multiple ways you can run Lighthouse, but the most straightforward is to run it as part of Chrome Developer Tool since it's already built in. Just open up Developer Tools, click the Lighthouse tab, start the audit and wait for your scores. Lighthouse also has a browser extension, a CLI tool that you can use from the command line or programmatically, and there is also a Cypress plugin based on Lighthouse which you can easily use with your Cypress test. Cypress is currently one of the most popular testing frameworks out there, and one of the things I like about it is its ability to cover other types of testing easily. When I found out that there was a plugin to integrate Google Lighthouse with Cypress, I decided to give it a try. The custom plugin created by Marvin Freshey leverages your existing Cypress test and introduces additional commands to check the performance of any pages.

Now this doesn't mean that you should use Cypress for all your performance testing needs. There are other tools that are much equipped to do that, especially for load testing, but what the plugin offers is a way to easily track your Google Lighthouse scores by extending Cypress default functionalities. Anyway, enough theory. Let's start with a hands on session and I'll show you how to set up Cypress Audit and run your first performance test.

To actually see this in action, I plan on having a link in the show Notes for this episode with the video of this recording, so I have.

Prepared an example GitHub repository that you can follow along if you get stuck on any of the steps that I'm about to show you. So feel free to clone this repo, but if you can, don't look at the solution right away. As a prerequisite you need to have Node and NPM installed. I've added the installation link here, so if you haven't got Node in NPM yet, please pause this video and download the LTS version, you would also need a text editor of your choice. Personally, I like to use Visual Studio code, which I'll use for this session. Feel free to install Visual Studio code or just use any text editor that you prefer. I've switched to my terminal of choice here and this step is completely optional. If you already have an existing Cypress project, you don't have to do this bit. However, to those who wants to set up a project from scratch, then stay tuned. So first I will create a new directory that I will call Cypress Lighthouse Project. To do that, we just need to type the command mkdir and then our project name. Next, let's change to this directory and then type npm init y. If you're not familiar with npm, we are simply initializing this new project and as a result it will create this package JSON file which we use to manage our project dependencies.

The next step is to install Cypress and Cypress Audit. To do that, let's type npmi d followed by cypress at 6.2.0 and then cypress audit at 0.3.0. Now, there's quite a few new commands here, so I will explain it one by one. NPM I is a shortcut for NPM Install. The D stands for Dev dependencies. We are simply saying that we want to install Cypress version 6.2.0 and Cypress audit version 0.3.0 to our project's dev dependencies. The reason why I've chosen to hard code the version numbers here is so that I know that this demo will still work in the future. The installation might take a while, but once it's finished you should see an output that both Cypress and Cypress Audit have been added successfully. I've opened the project on Visual Studio code, and the next step that we need to do is to initialize Cypress itself. If you're not aware, Cypress comes predefined with a default folder structure that we can extend easily. To initialize Cypress I'm back on my terminal and I'm just going to type npx Cypress Open NPX is basically a shorthand notation to execute a specific command. Right now I want to execute Cypress Open, which will open up the Cypress Visual Test Runner. Once that's finished, you can see that there is some example files that Cypress comes with, as well as other predefined folders, which I will explain briefly later on. You might notice as well that a JSON file called Cypress JSON is also created. It's currently an empty object, but we will modify this later on. For now, let's Go ahead and delete the example test files as we don't need this on this demo. What I'll do though is I'm going to create a new test file that I will call Cypress Lighthouse Spec JS Just before we set up Cypress Audit, I just want to quickly explain the other folders and files that's built in within Cypress, because we're going to make some changes to some of them. First we have the Fixtures folder. A fixture is simply a way to load a fixed set of data that is stored in a different file. We won't be using Fixtures today in this demo, but it's good to know what they are are used for. So next we have the Plugins folder. Plugins allow you to extend and modify the default behavior of Cypress. This is where you can set up plugins such as Cypress Audit so that the methods that you need to use to run the performance audits can be exposed. Finally, we have the Support folder. Files that are added in the Support folder are run before every single Spec file in Cypress. It's also a great place to put custom commands in this folder for any reusable commands that you want to be made available to all your Spec files. Before I show you some code on how to configure Cypress Audit, let's have a look first at their GitHub repo. So according to their readme, there's three steps outlined on how to set up Cypress Audit. The first step of installing the dependency, we've already completed this. The next step is to prepare the server configuration. So looking at this, there's a fair amount of code that we need to add. But rather than just copying and pasting this into our plugin index JS file, let's look at what's happening here first. So by default, when you use Google Lighthouse from your command line, it opens a new web browser. When it does the audit, I'll show you what I mean.

Let's just Quickly check the GitHub repo for the Lighthouse CLI. To install Lighthouse CLI, it's quite straightforward. Just type npmi g. Lighthouse g stands for global, so you can use this globally on any projects. I've already installed Lighthouse CLI on my machine, so on my terminal I'm just going to type Lighthouse followed by the page that I want to check, which is Airbnb. Let's just wait for a couple of seconds for it to load. And you might notice that when running this command, a new Chrome browser appears as it runs the audit on the Airbnb page. Now at the same time, if you're already familiar with Cypress, Cypress also launches its own Chrome browser when we run our tests. Because we want to run the Lighthouse test inside the same browser as Cypress, rather than opening a new one, we need to add this configuration code here.

Great, we're getting there. Now I'm back in my Visual Studio code and I'm going to start adding the code to set up Cypress Audit. I'll first remove the comment so you can easily see what I'm about to type. Now, Cypress actually reacts to various events. In order to hook into a specific event and perform certain actions, we need to use the on function. The first parameter that this function accepts is the event name, which in this case is beforebrowserlaunch. Next, this particular event will output two objects. The Browser object, which gives us information about the browser, and then the Launch Options object, which gives us information on how the browser is launched. For now, I've just set browser as an empty object because we won't need this for the Cypress Audit configuration.

Now, to show you what values the Launch Options object contains, let's simply log Launch Options. For example, let's open up the Cypress Test runner again by typing NPX cypressopen and then select our spec file that we have created already. Now, if we go back to our terminal, we can then see an array of command line arguments that will be passed to our browser. I highly recommend reading the Cypress documentation about the browser launch event so you have more information as to what else you can do with it.

Let's continue with our setup. The next thing that I'm going to do now is I'm going to import some functions from Cypress Audit. To do that, I need to require Cypress Audit first. Then I need to get the Lighthouse and Prepare Audit functions.

Next, we need to pass the Launch Options object as an argument to prepare Audit. So we can pass into Cypress Audit the command line arguments that we are using to launch the browser. Then finally, we need to create a tos. A toss in Cypress allows us to execute something on the Node JS backend process. The Lighthouse CLI tool is a Node JS process, so in order for us to invoke this function, we need to wrap it as a toss. I'm just giving the task name of Lighthouse as well to make it simpler. The third and final step that you need to do in order to complete the Cypress Audit setup is to simply import the Cy Lighthouse command to our Support Commands JS file. This is to make Cypress aware of this new command.

Now we are ready to write our first test, you can do the audit on any website of your choice. But for this demonstration, I'm going to run it on this website called PhotoBiz.

Let's navigate to the spec file that we have already created and then construct our first test. To those of you who are not familiar, Cypress is actually built on top of Mocha. So the describe syntax as well as the IT syntax that I'm using here is actually coming from Mocha. Let's visit the application first by calling in SAI visit and then pasting the URL for PhotoBiz and then to actually run the audit, let's type SAI lighthouse. Notice that a new tab is open within our Cypress Visual Test Runner and it actually opens the photobiz website just like the Google Lighthouse CLI would. Once the audit is finished, you can see the results of the test here. Now, by default, Cypress Audit will run the test based on a score of 100 for every metrics. You can of course override any of the metrics and thresholds depending on what your requirement is.

Let's add another test. However, this time let's modify some of the thresholds. Now, let's be honest, most websites will not have a 100 perfect lighthouse score. It might be that you're just starting out to monitor your metrics and then you decide as a team that it's better to set a custom threshold as a baseline.

Now, rather than typing in the website URL again on the second test, I'm just going to quickly move the photobiz URL to my Cypress JSON file and then set this as my base URL. Cypress will know that this URL will be the base URL for all of my tests going forward. Then when we call SAI Lighthouse, we can actually pass in a custom object configuration. We can modify default properties like performance and then set a custom score of 50. For now, let's do the same for accessibility. Again, I'm just setting a custom score of 50. Then for SEO, let's set it to 70. Now let's go ahead and save the changes. Once I press runs our test, it will pick up this custom Thresholds and notice that now our test is passing because the actual scores are now within the custom thresholds. If I also quickly just open up developer tools and inspect the arguments that we passed to SAI Lighthouse function, you can see here that it contains the thresholds that we have specified.

You might wonder what other metrics can you override. If you navigate back to Cypress Audit's Readme page, there is a section at the bottom, listing the categories and audits that you can customize. Apart from performance, Accessibility and SEO, you can also customize the scores for Best Practices and PWA or Progressive web apps. Generally speaking, the higher the scores, the better the performance. You can also customize specific performance metrics by setting the score in milliseconds. Now, before you customize this, it's a great idea to know what some of these metrics are. The latest version of Lighthouse 6.60 and in this latest version there are six important metrics to understand. The first one is First Contentful Paint or FCP. FCP measures the time it takes for the browser to render the first piece of DOM content. This doesn't mean that the user can't see anything yet on the screen. Large ContentFullPaint or LCP, on the other hand, measures the time it takes for the browser to render the largest piece of DOM content, such as an image or a text block. The Speed Index measures how long it takes for the content to visually load while the page is loading. Time to Interactive or TTI measures how long it takes for the page to become interactive so the user can start using it. The Total Blocking Time, or tbt, measures the time the page is blocked after responding to a user input, such as button clicks. If a toss executes for more than 50 milliseconds, anything above 50 is considered a blocking toss. Finally, we have Cumulative Layout Shift, or cls. CLS measures how often the page layout shifts while the page is loading. Have you ever experienced visiting a website and clicking a button, but suddenly the whole page shifts downwards because additional content got loaded? It's annoying when that happens, right? So anyway, with all of these performance metrics, the the lower the scores, the better the performance is.

Now that you have a brief introduction about these different performance metrics, let's try and customize some of them in our test. To do that, simply type the metric name. For example, I've typed First Contentful Paint, replacing the spaces with a hyphen, and then setting the score in milliseconds. Let's do that for a couple more metrics. However, feel free to select a different run from what I'm setting here. Once we save the changes and we go back to the Cypress Test Runner, you'll be able to see which performance metric failed to meet the threshold. The last thing that I want to show you is how to run the Lighthouse test on Desktop Viewport. By default, Cypress Audit will run the Lighthouse audit on a mobile viewport. If you want to override this, then I recommend reading this documentation from Google Lighthouse to understand how to configure it. So if you want to run it on desktop, the first property that you need to set is form factor. This needs to have the value desktop. Next we also need to set screen emulation. You can either set the width and the height, for example, but if you want this to be disabled then you just have to set the value of disabled to true. Let's quickly add that to our test. To do that I'm going to pass in a new object because this is actually not part of our copy custom thresholds. So on this new object, let's set form factor to desktop and then screen emulation to have the object disabled. Set to true. Now when we save our changes and run the test, Lighthouse will now audit the desktop layout instead for some final refactoring and tidy up, I'm going to create a beforeeach hook and move cy visit so we don't have to call this twice. Next I'm just going to create separate variables for the custom thresholds and the desktop configuration just to make it more readable. And there you have it. I have shown you how to use Lighthouse via Cypress Audit within your Cypress test. As a reminder, you can use my GitHub repo as reference in case you get stuck with any of the steps. I hope you found this session useful. And just to wrap things up, I talked about the importance of performance testing and how tools such as Cypress and Google Lighthouse can help you monitor the performance of your application every time there are new code changes from your team. I then showed you how to set up Cypress Autodes from scratch or adding it into an existing Cypress test framework. Afterwards, I explained the various commands needed to run your first performance test and an explanation as to what the different metrics mean. Again, thank you so much for tuning in to my first Automation Guild session and if you have any feedback, please do let me know. Thank you again and happy testing.

Thank you Marie for your performance and site reliability awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com P82 and while you're there, make sure to click on the Try Them Bolt Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance testing solutions, Load Ninja and Load UI Pro. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack performance testing awesomeness as always test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance and Site Reliability Podcast. Head on over to testguild.com for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.