Performance Testing from a User’s Perspective with Jacek Okrojek

Five, four, three, two, one. Drop that.

Welcome to the Test Guild Performance Podcast where we get together to learn more about performance testing and site reliability with your host, Joe Colantonio.

Hey, it's Joe. And welcome to another episode of the Test Guild Performance Podcast. Today we'll be talking with Jazzik Akrojczyk, all about performance testing from a user perspective and rapid performance testing. Jazzik is a test automation lead at Test Test Army, a QA and software testing company and he specializes in performance testing. He actually started his career, I believe, at Ericsson and Telecommunication and since then he's gone on to work in gaming, financial, medical, e commerce. So he has hands on performance testing experience in many different industries. He'll be sharing some of his insights he's learned along the way around performance testing as well as an approach he calls rapid performance testing. So listen up.

Let's face it, performance testing is tough. Traditional load testing tools create scripts that are bogged down in data that's hard to read and require a lot of work, even for simple playback. Load Ninja cuts out the dirty work by using AI to inspect and debug your code almost instantly. No coding necessary. No changing of hands or relays between teams either. Give it a shot. It's free and easy to try. Head on over to testguild.com smartbear and click on the link to learn more.

Hey Jazzk, welcome to the Guild.

Hi Zoe. Thanks for having me here.

It's awesome to have you on the show. I met you at Perf Guild last year. You did an awesome session on performance from a user perspective. So that's one of the things I like to talk about. But before we get into it, could you just tell us a little bit more about yourself that I may have missed in your intro?

I'm working in performance testing area, but not only I was working in test automation area, I'm a consultant. I try to help our customers in building test automation solutions. And of course I spent a lot of time doing performance testing. That's more or less my duties.

Very cool. So I guess the first topic I'd like to touch on is performance testing from a user perspective. Folks that don't understand that term, especially if they're old school performance testers. What is testing from a user perspective mean?

In traditional performance testing approach we try to focus on the server side performance. So we try to simulate the request that usually sent by the browser and we focus on the main request like a request that downloads the main HTML document. But this approach does not give us reliable Results because a lot of things happen when browser renders the web page and the time that user sees is much longer than the time that most performance testing tools reports and performance testing from user perspective tries to solve.

This problem and tries to provide the results that are seen by the user results from the browser and can tell us how fast user can see various elements on the website.

Are there any metrics you use or recommend for being able to get a user perspective info when you're doing a test?

Well, so every year we get new metrics. I remember when I.

Started performance testing from a user perspective, I think five years ago. Then we focus on the browser events like don't content completed or unload events and at that time those metrics were sufficient. But a lot of things happen in this domain. It seems that Due to latest JavaScript tricks, both metrics are not reliable anymore and we try to search for better metrics. There is a lot of interest in visual metrics as Speed index is one of such metrics. It tells you how fast browser renders entire content of your website and also tells you how the progress of the rendering looks like. And then on top of this you have a metrics that tells you when 80% of a website was rendered, 90%, 95, 99% of the website was rendered and finally everything was rendered by the browser. So those are metrics which can tell you how fast a browser renders the website. But on top of this we have metrics that comes from sociology approach to performance and it looks like not all elements of the website are equal and user tends to focus on the big and main elements on the site. So we have a metrics that can tell you okay when the first H1 header was rendered and metrics that can tell that tells you when the biggest picture was rendered. We call it the hero image and that metrics might also be useful when you analyze the performance of your website.

Now how do you collect these metrics? Is it all just exploratory? It's just running commands or what tools would you use to be able to collect these type of users performance metrics?

Okay, so I think that the standard tool for measuring performance from user perspective and measuring visual metrics is a webpage test. It is free software, Public Insta is running in the Internet. You can put the address of your website and start testing and you have several locations from which you can verify the performance of your website. And you can also create your own instance of WebpageTest and run it behind the firewall to verify your intranet Pages or pages that are under development. So I think that all testers who, who starts with performance testing from user perspective should look at this tool. Definitely. And there is also a book that describes how we can use this site and how we can use all the features of this software. So I really recommend it. And you can find a lot of, maybe not a lot of, but sufficient amount of podcasts and videocasts on YouTube that tells you how to use this tool and how to gain the most of it. So that's the first tool. The second tool will be Site Speed IO and here you need to download the software and run it from your instance or from your machine. But it's quite easy if you use Docker. You just download the images and you have a completely full setup. And the setup includes not only the software that can run tests, but also Grafana, which can collect and present results from several runs. And you can monitor performance of your site and do a lot of additional stuff like verifying performance of all of the pages of your site, or just selective pages simulate different network connections and much more. And you can find all the details in the documentation.

Are there any tools built into chrome like DevTools that can help people get quick and dirty measurements?

Yes, yes, yes, definitely. So Chrome Team, I'm sure if Word Wire is a good one. But now when you open DevTools in Chrome and navigate to audits, you can find the Lighthouse audit and you can start the audit tells you what is the performance of your site using simulating mobile or desktop version of a browser. So it's also really good tool to start with. Definitely. It's worth checking your Chrome and verifying what you can find there.

Awesome. So another trend I've been seeing for one user performance is people leveraging their existing functional tests, but then getting these performance data out of them. So that's, I think a lot of times people use Cypress because Cypress is able to hook into the Google Chrome measurements. So is this a trend you've been seeing or how do you normally see how people are able to run a performance test with just one user and then capture the data?

I don't use Cypress. I know that this tool can help you in your UI test, but I use actually Sites Speed IO and. Well, maybe because our customers comes to us and they are focused mostly on performance tests. We don't have customers who ask us for running functional tests and then on top of it a little bit of performance test. So maybe that's why I like sitespeed IO and with sitespeed IO I can also run some transactions using Selenium and I can tell sitespeed IO where it should start measurements and when should finish the measurements and then can collect the data that way. I guess Cypress is a nice option, but I never tried so far.

I guess another thing I think of when I think of performance is putting a load on the system and then measuring how the application behaves for one user. Is that still a valid approach or is that something you would recommend?

I think it's a.

Good point to start. When we run performance tests we usually have a term baseline measurement and that is the measurement that is collected only one user. For me it is the best performance that you can get from the site. And then once you add some synthetic load, you can check how this performance change and how it behaves. So I definitely recommend starting with one user. And in many cases even when you have one user you can spot some issues, big images or just some content that might be smaller and be downloaded much faster. So starting from one user is actually good practice.

In your presentation you gave at Perf Guild you were talking about something called, you talked about the DOM Content Loaded Event and Peak Timing API. Are these still valid? Do you still use these? And what are these events? I guess is my question, how does it help capture one user performance data?

I talk about them because it gives you some context where performance tests begins and how it evolves. And still.

If you are new into performance testing, you might find some materials which advise you to actually check those metrics. I think that today we have much better tools. The visual metrics that I talked about earlier I think are much more reliable. Even the audits that are built into Chrome gives you more interesting data. So it's good to know that those metrics still exist, that you can fetch them, you can use them in your functional tests when you would like to wait for some events or some resources to download. But when you run performance tests, I think you should focus on newest metrics.

Jazza, can we talk a little bit more about the metrics you can collect with dev tools when you're doing one user performance testing?

Yeah, so actually guys from Google has slightly different approach to performance that you can see in the tools like WebpageTest or Site Speed IO. So what you can get from the DevTools is the time when the first contentful paint was made.

It's a moment that you can see some initial content on the web and also the metric that is called first meaningful paint that allows you when something bigger content was presented by the browser. So so those metrics are important for.

Guys from Google. And it makes sense because some research shows that actually the perception of the performance comes from the moment when you can see something on the screen and then you think that okay, something happened and it's efficient for you. And then you have a time to interactive metric that tells you when user can start interaction with the page. And it also, it looks like this moment is also important for the user and it also affect his perception of performance of a site. And actually when you try to compare several sites, you can see the significant differences in this area. And some developers optimize for seeing huge content and some optimize for time to interactive. And Airbnb for example, optimized for time to interaction. So you can really fast for some accommodation. Even you don't see any nice pictures in the background. That's done on purpose. And on the other side, first you can see some content and maybe read or look at them. And then at some point you can start interaction with the site. And it also make a little bit difficult to compare sites because some authors optimize for for content and some will optimize for interaction.

Well, it's quite difficult to compare such.

Sites besides one user. Performance testing Another topic I think you've been really talking more about is rapid performance testing. So I'm not even sure what that means. So what is rapid performance testing?

Well, it's a concept, it's a hypothesis. At this moment.

I even looking for a good name for this concept. I selected the statement rapid performance test because.

It tells us that it's relatively fast, but it's not shallow. I'm still searching for the right name of this hypothesis. I selected the word rapid because it means fast, but it also means that at least for me, with the right attention to the details. And.

The idea comes from my experience, I think two or three months ago I collected results from my performance tests. I think I collected results from 20 projects and I tried to analyze them and see what are the main reasons of the performance problems. And I was actually interested in finding a way to optimize the effort to find performance issues and maybe minimize the cost of performance tests. And what I discovered was that.

In 70, maybe even more 70% of cases I could discover main performance issues by running just one not very complex scenario. And then it gives me the idea that maybe it could be a sort of a good approach to recommend actually running just one simple scenario. Initially it's quite cheap, it doesn't require a lot of knowledge about performance testing. And if Your tests are fine, or if you get the satisfying results, then you're fine. And most likely your performance, the performance of your application is sufficient. And if you have, if you find some problems, then you can invest in more sophisticated and more detailed performance tests.

So that's the idea. Actually. The idea is run simple test, simple performance test and collect the results. And if the results are fine, most likely your site has or your system has sufficient performance. And if not, then you need to invest a little bit more time and resources to find the problems.

Cool. So it seems like a quick smoke test almost. If it passes, then maybe you can decide based on how critical that application is, that your performance is good enough. If not, then drill down more.

The problem that I would like to tackle was that usually when customers would like to run performance tests, they specify several scenarios and quite complex conditions. And then when we estimate the cost and time of such tests.

They thought, okay, that's too expensive, we don't want to run such tests. We will go live without any tests.

So I thought that maybe we can do something that will give us some really useful information, but it will not cost us as much as running full scope of performance test. And you, in my opinion, running just one simple case will give you sufficient confidence in your system. Because as I mentioned earlier, most of the issues related, most of the issues that are discovered during performance tests are related to infrastructure configuration problems. And you can discover those problems just by running one scenario and you can really quickly solve it. And you don't need to invest a huge amount of money and time in performance testing.

So can you give an example? Because there's so many metrics and things you can monitor during a performance. It's like, how do you know what's acceptable or what is not or what's sufficient? What are you looking at to say, okay, this test was sufficient because X, Y and Z didn't raise a red flag.

So when it comes to metrics that we analyze during performance tests, we usually select two metrics. It's a number of concurrent users that can that application can support and maximal response time. Because when you increased number of concurrent user, usually the response time increases as well. So looking only at one of those metrics, looking only at the number of concurrent user, metric cannot be sufficient because the response time could be quite huge and actually not acceptable. So we usually focus on the number of concurrent user and response time that is.

Served by the server.

Okay, Isaac, before we go, is there one piece of actionable performance testing advice you can give someone to help improve their performance testing efforts and let us know the best way to find or contact you.

I think.

My approach is well start doing performance testing and and as I mentioned earlier, just if you don't have time for running extensive performance test.

Run performance test with one simple five step scenario and check if your service survives and give you sufficient performance and hence in most cases, at least now at this moment I believe that it will.

Significant improve your.

It will. Oh man, it will.

I believe that it will.

Give you a lot of benefits and in this way you can avoid most performance issues or actually find most important performance issues. So my recommendation is start doing performance test even simpler. You can find me. Well actually I'm not.

A huge fan of the Twitter so you can contact me via my email address and it's jacek.acrylicestarm.com yeah, it's the best way.

Thank you Jaq for your performance testing awesomeness. For the second value we covered in this episode, head on over to testguild.com P07. And while you're there, make sure to click on the Try It Today link under the Exclusive Sponsor section to to learn all about SmartBear's performance testing solutions like Load Ninja. If this episode has helped you in any way, why not go ahead and rate and review it in itunes? Ratings are extremely helpful and greatly appreciated. They do matter in the rankings of the show and I do read each and every comment and they also help make the show even better. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe and my mission is to help you succeed with creating Full Stack Automation Awesomeness. As always, test everything and keep the good. Cheers.

Thanks for listening to the Test Guild Performance Podcast. Head on over to testguild.com for full show notes, amazing blog articles and online testing conferences. Make sure to subscribe to join the Guild and continue your testing journey. This has been a Joe Calantonio production.