Front-end Performance User Experience Testing

Welcome to the Test Guild Automation Podcast where we all get together to learn more about automation and software testing with your host, Joe Calantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Automation Podcast. Today we're going to be going over what I think is a really important topic for testers and that is front end performance. Don't be scared by the performance aspect. You don't need to be a performance testing engineer in order to do these type of activities. I'm actually going to go over a blog post I recently wrote called the Complete Front End Performance Testing Guide. I have a link for that in the show Notes Going over a session we had at the 2020 PERF Guild Conference with Andy Davies, who is a performance testing professional. So I'm going to go over what his session was about. Some tools that can help you, Especially as we head into Black Friday. You're going to want to make sure your applications are ready to handle the increase load that they're probably going to experience during the holiday season. And if you haven't already, I just wanted to let you know you have one week left to get a special early bird discount on the Automation Guild tickets for 2021. Head on over to GuildConferences.com and register for the 5th annual online conference dedicated 100% to helping you succeed with all things automation testing related. Our goal is you learn a tip, tool, technique or best practice that's going to help a new or existing automation project and set you up for success in the new year. You definitely don't want to miss it. We have Simon Stewart, who is the creator of Selenium, joining us and it may be one of his last sessions he gives at an online conference. And also we have the creator of Appium. We have a bunch of awesome sessions so you definitely want to check it out. Head on over to GuildConferences.com Register for that right now while the early bird ticket sale is going on.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Their cloud based test platform helps ensure your favorite mobile apps and websites work flawlessly on every browser, operating system and device. Get a free trial. Visit testguild.com sourcelabs and click on the exclusive sponsor section to try it for free for 14 days. Check it out.

All right, so as I mentioned, as we head into the holiday seasons, the topic of front end performance testing of consumer facing businesses like retailers, publishers, financial companies, all types of companies are going to become even more critical. Also, I'm sure most folks probably aren't even planning for how to handle higher traffic than normal on their sites in this COVID 19 world we're living in. So I expect an increase in traffic on sites as people are more reluctant to go out and shop in public. They're going to do more and more things online. So I think this holiday season we're going to see even bigger issues with performance. So you want to be on top of it before it affects your application. And I know this can be very frustrating. Our teams might have initially put in a lot of effort into making our sites wicked fast, but one of the challenges you're probably facing is how to stay on top of things once your application is released in the wild. So you have multiple probably iterations, multiple things and patches that have been released and over time you probably lost track of how performant is actually our application for our end user. You know, I speak with a lot of different, you know, many developers and testers who struggle with things like how to monitor their site speed over time and how to know what a new release of a code or what a new releases code change will have on the front end performance of their application. So what I've done is I've actually taken the advice I found from many performance experts I've actually interviewed on my if you haven't listened to it yet, the Test Skill Performance podcast. Once again, I have a link for that in the show Notes, but a lot of this is actually taken from Andy Davies, who is the Author of Using WebpageTest Web Performance Testing for Novices and Power users and a 2020 PERF Guild speaker from this year. He did an awesome session around this topic. And a lot of what I'm covering in this podcast episode is based on Andy Davis's work. So let's get into it. The first thing that Andy pointed out was, or you need to be aware of, also is the psychology of webpage response times. And so let's start a little bit with human psychology. I think if you step back maybe 50 years ago to around 1968, there was a guy called Bob Miller who was researching how we respond to delay. And he found that as long as you receive a response within a tenth of a second of an action, you perceive that as being instant. And if you press a button and a light comes on in 100 milliseconds, you receive that as instant. But as the delay creeps up around a third of a second, you begin to notice the light if it's on or off. And if you receive a response within A second you can seamlessly carry on and it doesn't interrupt your flow. So that's good. But the longer that delay becomes, the more likely you are to what we call bounce. And what I mean by bounce is when you go to a website and it doesn't respond quickly to you, you go directly off and you go at another website. That's bad because obviously people aren't spending time on your website. If it's an E commerce site, they're not going to buy anything because they're not spending time on your site. So bounce rate is very important. And in 1968 they found that that limit before someone bounces or loses interest is around 10 seconds. And I think a few years ago actually Microsoft did a similar research and found the limit was more around 7 to 8 seconds mark. So it's getting even less time that people are going to stick around on your site as your page is loading. So user response time from a front end performance perspective is very critical. So when we talk about performance, a lot of people are thinking about stress testing and load testing and that's critical as well. But that's not what we're talking about here. Even though that's what actually ultimately probably impact the front end performance of your user. What we're really talking about is how your user interacts with your application one at a time on their desktop or from a browser, from a mobile device. And you might be wondering, well, why does application front end performance response time even matter? To me, I'm a tester, what do I care? Well, maybe cynical, but everything I think comes down to money. And money. So if your application is not responsive to your end user and so your company is losing money eventually, if your company's losing money, guess who's going to be impacted? You. If you're an employee, they're going to start, you know, a lot of times they'll just go, oh, we're going to cut headcount in order to bring up our share price or something. And the easiest way to do that is to get rid of people. So you know, in my view, the more performing application is less likely that is going to happen. And so you definitely want to be aware of this. It's not one of the only issues, but money is one of the leading factors and also just something to be aware of. So if you make people wait on your website, if you deliver a slow experience, it's fundamentally gonna have a negative, it's fundamentally gonna have a negative implication for your business. And you can actually see this when you look at real data of the experience people get on websites and how that influences their behavior. So once again, I'll have a link to this in the show notes, but there's a chart that Andy gave up called how speed affects a user users of behavior on your website. And in the chart, people with faster experiences view more pages on a website. Right? It sounds kind of. Yeah, makes sense. But a lot of people don't think about that. So if you're a retailer, an E commerce site, that means they look at more products, which probably leads to more sales. And if you work for a publisher that relies on advertisement, that means they're going to be fed, they're going to read more stories, which then you can then give them more ads. So you're getting a benefit because the speed is encouraging them to stick around on your site more and interact with your site more, which ultimately is going to lead to better behavior that encourages the way you make money for your company. He also shows in another slide how bounce rate affects your website as well. And so bounce rate, as I mentioned, it represents how many people come to your site, visit one page and then leave. So a high bounce rate means that folks are leaving your site without taking any action, which is not good. And you see once again in this chart, hopefully you'll reference it in the show notes. You see the bounce rate is lowest at the 3 second mark, then it climbs after that. So the longer you make people wait, the more likely they're only to visit one page or they're going to bounce. It also shows in this chart a conversion rate. And the conversion rate is how many people spend money or bought things. And in the chart you'll be able to see that virtually nobody converts below three seconds. And that probably that makes sense, probably because you know, a few pages complete before 3 seconds and the longer you make someone wait, the less likely they are to convert. And in that four to seven second mark, that's really the sweet spot. And so in a 4 to 7 second mark, conversions rates drop from 5 to 4% and that translates into lost revenue. Andy also has an example of how he helped a retailer improve their site speed. And once again there'll be a chart that shows this. And so what they did is they targeted Android only users and he made some changes that improved the median experience for those Android visitors by 4 seconds. So that's all they did. They made that one small change and he said they saw the amount of revenue from those visits or those visitors increase by 26% just by making that small tweak just in time. So a lot of this is going to sound more dairy, but rather than practical. But I'm going to show you some tools that Andy shared with me that can help you focus on how to get that response time down. But before we dive into that, you know, I just want to talk a little bit more about how people currently are thinking about performance and why. I think there's some blind spots that a lot of people are going through as they're developing software. So your team, probably, as I know my teams did, they spend a lot of time, a lot amount of money on building and tuning your server farms and the databases and testing that capacity to ensure that you can really serve up that initial HTML payload really quickly to your site visitors. And that's critical. I'm not downplaying that, but that's like a majority of all they focusing on. And you know, if you look at some of the top sites and Andy showed some UK sites in his example, it showed that how long it took the back end to generate the initial response. And so in the chart you see these little pink lines and then you see a blue line for the front end. So if you can visualize this, the pink line represents the back end and that blue line represents all the other resources that make up the payload that you're serving up your end user. So that would be your images sizes, your scripts, your style sheets, all the things you need to complete that payload. When focusing on performance, it's essential once again not to ignore the backend because until the back end has a it delivers a response, there's no work for the front end to do. But in this chart you see a majority of the work that affects the actual experience. And that's what we're talking about, end user experience. It's actually happening in the browser. So in order to measure the front end performance and to get front end performance insight, you need a metamodel that's going to help you understand how the metrics you gather map to the actual business experience. And so Andy shared a mental model of the front end that makes up a majority of the payload that takes up the time that people are waiting around, waiting for your site to load. So it's a really cool visual Andy gave and once again, that will be in the show notes. And so the image you showed represents visual clues a visitor might have that things are working properly as your page is loading. And so he gives a case where the browser bar change the address to the website. But at what point does the page actually become useful. That's critical because if it doesn't become useful, then most likely the user is going to bounce if it's going to take too long. So you want to make sure that the first thing that appears, even if the site's still loading, someone could start interacting with your site at least so that they won't bounce. And that's going to be different for you based on what your application does compared to someone else. So it's different, sites are different. So for a new site, it may be when someone can start to read the news, that's going to be the first useful thing that your site does. You want to make sure that is loading as fast as possible. It also could be when a product image appears so the visitor can see that they're on the right page for a retailer. So you really have to consider at what point does your page become usable. And in the example Andy gave, it was pretty late because the menu button wasn't immediately available for the type of action that that user was expecting right away. So when you're thinking about front end performance, you're thinking about, you know, how long does that page take to load before it becomes useful. So you can do a lot of things where even though things are still loading up, as long as you have something happening first that can allow the user to start interacting with this site, even though other things are loading up, that's going to help your user experience as well. So you want to know how long it takes for your page that you're testing to become usable and what happens in that beginning phase. And there are actually two broad ways you can measure how a page performs. And so the two ways are synthetic and in the wild. So synthetic is in the lab style environment where you have defined test setups in a known condition and then you have in the wild. So in a real person's browser, using, you know, whatever phone they're using, connecting to whatever network they're using, and both approaches have their place. But this we're going to talk about here is going to be talking about in the lab approach, because that's the one that most closely fits how we build performance into our initial workflow. So it's critical as your software development lifecycle is going on, like anything you want to shift it left and that everyone's being involved so it's not too late in the process where the application's already developed and then you're starting to test in performance. You can't test in performance to an application that's not performant. Just like you can't test in automation to an application that's not automatable. Right. So you want to get involved in the software development lifecycle. The best way to do that then is through synthetic, even though in the wild has its place too. So if you're doing any type of crowd testing is really, I've seen helped a lot of, a lot of people as well. But we're going to just focus on the synthetic part in this episode. So how do you start thinking about front end performance if you do nothing else? Right. The critical takeaway from this episode is to stop building performance in your software development lifecycle. As early as possible is essential. And there's some situations during which you should be thinking about performance and those are like when you're still in the planning phases, think about how big your pages are and will that be, you know, what they're going to be composed of, how big they're going to be, because the bigger they are, the more it's going to affect how load, how much it loads. And when you're running a test on each build, you want to understand whether the build has made the site faster or slower. That's critical as well. I used to use a tool called Serenity and it wasn't like as detailed on exactly what was causing performance issues, but it gave me a time that this test took maybe, you know, a minute, right. But I could tell over time if it took three minutes, then we might have an issue. It gave like a really high level indicator or flag that, hey, we need to look into this because over time, traditionally this test only took a minute and now it's taken three. And it just gives you a place to stop being able to pinpoint that you may have a performance issue. That's just one quick example. Also, when you're tracking your releases and probably using external, you want to use an external APM tool to understand how your performance is changing now that you've released that software to the wild. And when checking your releases using synthetic and real user experience monitoring tools to understand what folks are experiencing in the wild, that's going to be critical as well. So you can adjust to that information and then take it. And as you're developing software or making iterations on your software, you can build that into your software development lifecycle as well. So if you decide the application is going to include lots of fonts, lots of scripts and lots of images when you release it into the wild and discover that users are experiencing slow response times, you should revise your choices and to slim your site down. So that'll let you know that, well, you know, maybe we should scale this back. You would just want to be on top of things and not be caught off guard. So being a little proactive is going to go a long way. Also, like anything with testing, integrating performance into a continuous integration, continuous delivery pipeline is a must to test against your performance KPIs to ensure that you're delivering a good performing experience to your users. So just like you integrate your automated tests into your pipeline in your unit test, you also want to start integrating these high level performance criteria as well so that you're not caught off guard with your performance as it's released into your customers. So let's talk a little bit about how do you actually start building performance into your software development lifecycle? So you're probably asking right now, okay, how do I begin building performance engineering into my developer's workflow? So first we're going to just take a look at a few tools that you can use to test performance in a synthetic lab type environment. And then you're gonna, we're gonna cover a little bit how these tools can integrate into your CI CD process. So you can ideally on every check in, understand whether you have a positive or negative impact on performance. And once again, when I think about performance, I think a lot of people think about, oh, it's this long big planning, performance engineering process that you have to put in a huge load. You need to use a lot of performance tools. What I'm going to talk about here is a tool that pretty much anyone can use to help them from a one user perspective understand their performance. And James Pulley brought up a good point that a lot of times when people think about performance, they think about multiple users and they have no idea even how their application performs for one user. So it seems common sense how does it perform for one user. But a lot of teams actually completely bypass one user and just start thinking how it performs for multiple users without focusing on this first critical step. And because it's just one user, it means you can start testing as early as possible, which as I mentioned is going to be critical. So the first tool you're going to use that you want to use is Google Lighthouse. So this is actually built into the Chrome dev tools. It has a Lighthouse built in. So if you actually use have Chrome installed, if you go to your web application, you right click on it and you inspect your web page, you're going to have a tab that allows you to experiment straight away and explore the features that the Lighthouse offers. So when you open up Dev Tools, you'll have an option for Lighthouse, it used to be called I think Audit Panel. And when you start to use Lighthouse, you can begin examining the pages performance. Also accessibility, which is a big topic nowadays as well, and some SEO best practices. You can also choose to have a test page in a mobile scenario where it uses an emulated mobile device that has a smaller screen size or slow down CPU or a slower network. Or you could test on a desktop device. Now all this is free, right? And all this is baked right in. All you need to do is go to a website, you right click on it, you click on Inspect, and then when you click on Inspect, you'll have a tab called Lighthouse. And then in Lighthouse you can actually generate a report. It gives you an option to generate a report and it'll identify and fix common problems that affect your site's performance and accessibility and user experience. So really, really useful tool they're definitely going to want to start using right now or show you developers right now. And like I said, it doesn't take any fancy tooling or fancy knowledge, right? Anyone could start doing this right away. So I think it's a quick, quick win for your team if you're not even using it right now. And another way I've seen people use these Google Lighthouse into the development process is to actually leverage their existing automated test to capture this data. And as I mentioned, one way I do that is I use a tool called Serenity. And Serenity gave me a relative way to track performance over time. But if you actually want to tap into Lighthouse, you can actually use their API. And one tool that comes to mind is Cypress IO. I think Cypress actually has a audit plugin that allows you to use Lighthouse. So as part of your flow you can actually tap into this plugin and as your test is running, you can actually capture Lighthouse metrics. This is awesome because your automated tests are probably already part of your CI cd. And just by adding this extra feature is going to give you this high level performance data as well. So that's wicked cool. Also I keep mentioning relative performance, so it's not diving into every single. Like if you're using a performance tool like loadrunner, a lot of times you call, you create transactions for every piece of your website. This one gives you more of what they call a performance number for the Lighthouse thing. So it's a, it's called a performance number, right? So it's, it allows you to, when it audits the site, it gives you this performance score. And this score is useful because you receive a metric. And metrics are great because you can track that over time. And so this could be useful when discussing performance with your company stakeholders. So you could say, look, our metric is a performance score that we got from Lighthouse is 20. You know, it could be bad, could be good. It all depends on your application, what it's doing. But what's great about it is you have a number at least, and once you have a number, you can least track against that number, and that's critical. And you can start using it for having conversations. And having conversations is awesome because people are usually driven by numbers. You can say, hey, look, our performance number we're going to use is 20. And over time, if it gets worse, we could start saying, hey, why did it get worse? How could we make it get better? So you can really use it as a simple way to use to measure, to use it to measure over time as to whether you're getting better or worse, or as a comparison against your competitors. Like I said, because Lighthouse is built in, you can actually go to your competitor's webpage, right click on it and see how they do compared to your website. So that gives you a competitive edge as well. Also, what's cool about Google Lighthouse has other features besides the main performance score. There are the metrics that you can use to capture the visitors experience that they might have when a page is loaded. The three main ones you can use to dive into is first, contextual paint. And this is basically when did the content start to get painted on the screen. They also have one called Speed Index, which measures how long it takes for the visitor's screen to go from a blank screen to being fully complete and stable. The faster the better, obviously. So this is a great metric to capture that, to know when it comes to the experience of the user, how fast that speed index is, and also time to interact, which tries to measure when a visitor can start interacting with your application. Which goes back to what we talked about earlier. How soon can your application become useful? Because that's going to help determine whether or not a user bounces off your site or not. All these once again are metrics captured directly within Google Lighthouse, a free, completely tooled that you can start using right away with in Google Chrome. And you can help use these metrics to start judging your visitors experience. So once again, you can use that total performance score that, that the audit gives you to see at a greater, at a higher level, the metric you can use to track relative performance over time. But then you can also go deeper and use these lower level metrics to track and see where you're getting better or worse. Also, Lighthouse also generates some suggestions on things you could do to make things better. It also has a diagnostic piece you can use to see some reasons why you received that performance score that you did. And overall, Lighthouse gives you a awesome top level metric to track and sub metrics you can use to understand how to make your page faster. Another tool that kind of does this, that uses Lighthouse behind the scene that you might be aware of as well is Page Speed Insights. So I'll have a link once again to this, this article that I wrote that covers all this. I'll have a link to all of them as well. But Page Insight also gives you a little more detail as well. You should definitely check that out. So some of the tools that Andy brought up are not free tools, but stay with me, they're not free, they're paid. But these are just some tools. There's lots of tools. I'm not saying use these tools, but I think I'm gonna give you a reason why having some paid solutions are critical, whatever they may be. Let's just go over the ones that Andy recommended and you could use whatever, but I'll go over why Andy pointed out paid solutions. So the first one which is actually not not that expensive, it's called Debug Bear and actually help you track Lighthouse score over time. And it also produces some dashboarding. And I'll have an image for this in the post that I'll have a link to that shows you this dashboard and it just makes it really easy to see how all your key front end metrics are changing over time. He also pointed out another one called Trio, which is another product that you can use to create Lighthouse dashboards for you as well as track performance over time. Trio gives you a snapshot of the scores, timings from the latest test and the main score you can use to scroll that you can use as. Yeah, so Trio gives you a snapshot of the scores and things from the latest test and your main score. And so these tools are great to give you to really leverage Lighthouse not only to make a snapshot but also to be used as part of your SDLC to track how performance is changing over time. Another tool to check out is WebpageTest. This is a free tool and this is often referred to as the Swiss Army Knife performance testing tools. So it uses real browsers, not just Chrome or Lighthouse. But you could test it in Firefox, Edge, Chrome and other browsers, and you can test on real mobile devices. So another tool you can check out is called webpagetest. Webpagetest is often referred to as the Swiss army knife of performance testing tools. It uses real browsers, not just Chrome, like Lighthouse does. But you can also test against Firefox, Edge, Chrome and other browsers. You can also test on real mobile devices. And similar to Lighthouse, you can basically test through multiple locations around the world, whereas WebpageTest has four areas. And, you know, for the paid solutions like Debugbear and Trio, they have about 10 or 13. So obviously the paid options give you more, more locations. Also, what's cool with things like WebpageTest, it has a lot of options, including a capture video, which is really handy. After the test is complete, it generates a report. You receive a set of metrics about the page that are slightly lower level and more in detail than lighter hosts. One of the first things Andy pointed out with the webpage test is something called the filmstrip view that he said he uses all the time. It's very effective at showing key performance areas because everybody basically understands a filmstrip. You know, it builds. And he said it actually helps build empathy about what the visitor is experiencing because you could see over time what they're seeing and then you could kind of relate to a wow, you know, after five seconds they still can't interact with the site. I would be frustrated. I can empathize with them. What can I do to make it better for our end users? So WebpageTest gives you a richer, wider set of metrics than Lighthouse. It also gives you the ability really to drill down and test, to glean more information and actually understand why you got the results you did. So similar to there being commercial variants of Lighthouse, there's also commercial equivalents to web page test. So I mentioned WebpageTest is free, but there's also paid options. There's something called Speed Curve, which is the same edge as web page tests under the hood, but also tracks performance over time. And you can set it up to send alerts hourly or daily. And another one is called Caliber. So both of these will help you track your site's performance, whether in their staging environment or a real or live environment or over time. So why do we mention these tools? These are Debug Pro, Debugbear, Trio, Speed Curve. And the reason why, as I mentioned earlier, hopefully you stuck around, is because they all have APIs. And why are APIs critical? When a product has an API, it allows you to easily start testing or evoking a test on demand and get the results back when it completes. Which means you could then integrate it into your CI CD pipelines from Jenkins. Making API calls is very easy. So when you have a paid product and they offer these APIs, you can easily over time start integrating them into your CI CD pipeline so you can make changes. You could track how those changes are affecting your squad of Lighthouse or your absolute raw timings and webpage tests. So it's just really critical because you can actually start integrating these insights as part of your team stat CD pipelines and the paid solution allows you to access this functionality. And Andy mentioned building performance testing into your continuous integration cycle is in his experience the best way for you to stick to good performance practices. Because obviously if your team is getting feedback after every check in, they're going to be more aware of how they're coding or how they are developing. Software is impacting the performance and ultimately the end user experience, which then impacts the money or the value your application is giving to your users in the wild. So if you actually want to see the demo that Andy gave at the Perf Guild once again, just go to the show notes of this episode, which I'll have a link where you can actually get the actual session, actually see this in action. So some parting words of wisdom. Like I said, I always ask someone as you know, if you've been listening for a while One piece of Actual Advice Andy's one piece of actual advice is, you know, at the beginning of this episode we mentioned the massive improvement to the user experience that Andy got from the response time in the Android retail example. But he said it's not always that simple. So often, you know, you make a series of small incremental gains that eventually add up to larger ones. And the same is true for building front end performance into your workflows. You know, he highly recommends you start simple. Start perhaps with the page Speed Insights, Lighthouse or one of the commercial services, set some limits around where you currently are and use that to make things to help educate your team to see over time are things getting better or worse. When you make changes then over time you can start working on more advanced speed improvements. So just having that high level metric performance metric once again it can help create more conversations and you have a number that you can actually go to and say is it getting better? Is it getting worse? It's just an indicator. It's not the end all be all of performance, but at least gives you something to start with and just having that one performance metric that the Lighthouse gives you is an awesome starting place. And also remember, Google is planning to start using performance as one of the ranking factors in their search algorithm. So if search is important to your application, this means that your company is really going to want to be invested in the front end performance of your application as well. Because obviously you want your site to be ranking high in the rankings for SEO when people do Google searches as well. So hopefully you found value from this episode. All right, so I mentioned a lot of things in this episode, so in order to get it, all you need to do is head on over to testgill.com a328 and while you're there, make sure to click on the Try it for Free Today link under the Exclusive Sponsor section to learn all about Source Labs awesome products and services. And as I mentioned, if you're listening to this on the day this has been released, you have one week. One week for the Early Bird Specials. What's going to be the biggest Automation Guild ever? It's the fifth annual online conference dedicated 100% to automation testing. We've been doing this since 2017. It's not taking advantage of COVID 19 as you see with all these other events going on. We've been doing this for over five years. We have it down to a awesome system that our attendees seem to really like. Find it very engaging. If you haven't checked it out already, head on over to GuildConferences.com and definitely register for Automation Guild 2021. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack Automation awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Automation Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.