Playwright Performance Testing with Katarzyna Kmiotek

Get ready to discover the most actionable end to end automation advice for some of the smartest testers on the planet. Hey, I'm Joe Calantonio, host of the Test Guild Automation Podcast and my goal is to help you succeed with creating automation awesomeness.

Hey it's Joe and welcome to another episode of the Test Guild Automation Podcast. Today I want to share with you an episode I did a few weeks ago on the performance and site reliability podcast that I have all about Playwright and how to use it for performance testing, which I think you're really going to get a lot of value from. And just so you know, Automation Guild 2023 is coming up. I'm just about to announce the lineup for our annual online event that happens every year. You don't want to miss it. To find out more about all types of automation topics, not just functional automation, but also performance topics and more about Playwright Selenium, all the things that you need to succeed with automation. Head on over to automationguild.com and register for the early Bird now and today in for a special treat. We have the hot topic of Playwright here and Cat is going to be joining us talking all about load testing with Artillery and Playwright. So I'm sure we'll touch on a little bit about functional automation as well, but mostly hopefully how you can actually augment your Playwright scripts with Artillery, which I'm really excited about. If you don't know, Kat is a quality engineer at Zoopla, enjoys exploring capabilities of Playwright as a functional and non functional test automation tool. So a lot of end to end testing, accessibility testing, she knows everything about visual regression performance so really excited to have her on the show. I actually came up with this topic by seeing her blog post on this exact topic which I'll have link for in the show notes so you don't want to miss that. And her most recent post actually explores Artillery AIM Playwright integration that allows testing browser and service performance. So Cat volunteers also as the Code Bar Organizer, which is a coding charity for supporting people with under represented groups and learning tech. So really excited that she's sharing all our knowledge here as well. So you don't want to miss this episode. Check it out.

The Test Guild Automation Podcast is sponsored by the fantastic folks at Source Labs, the cloud based test platform that helps ensure you can develop with confidence at every step from code to deployment for Every framework, browser, OS, mobile device and API. Get a free trial. Visit testguild.comSourceLabs and click on the exclusive sponsor section to try it for free today. Check it out.

Hey, Cat. Welcome to the Guild.

Pleasure to meet you and happy to be here.

Yeah, great to have you. Really excited about this topic like I mentioned, I guess. Before we get into it though, is there anything I missed in your bio that you want the Guild to know more about?

Maybe the fact that I'm a career changer is fairly important for the role that I do as organizer, co organizer of the Code Bar chapter, Edinburgh and Glasgow, I try to promote that sort of way of thinking. There's no wrong path. I decided to do career change fairly recent and I cannot imagine my life without tech at the moment. So that's what I try to do in my spare time. Convince as many people as possible, especially female. That is great fun.

Awesome. Awesome. So what did you do before then, when you mean career change?

I was working in hospitality roles mainly.

Nice. So how did you get into tech then? That's interesting. So you were in hospitality. How did you even say, oh, I'm going to go into tech all of a sudden?

Yeah, that was big dive. That's true. I had my partner who was supporting me all the way and probably he believed in me more than I did that I can do this transition. So.

Nice.

I try to be that sort of support for other people as well, if possible.

All right, so I guess the next question I'm already off topic is there's a lot of things you get into with tech. So how did you fall in then into testing?

So did the coding boot camp. That was like teaching you as a full stack developer. And the area that I was most interested was like deployment, pipelines, everything that happened behind the scene and then the opportunity as an automation engineer peered right after the coding bootcamp and it was great experience because I do what I like. Being between front and back end some more DevOps.

Great. So this is not scripted. So once again, another question just came to my mind. A lot of people actually asked me about boot camps. I've never been through one, so it sounds like it actually worked for you. So do you recommend that as a valid way for people to maybe make a transition into tech and testing?

Yeah, especially for the people that they're not sure where to start because bootcamp provides you with sort of guided path. You know, you learn git and so on, so on in the right order. So that's benefit. But here there is like, let's say Code Bar is a charity that supports people who are not that lucky that can afford the bootcamp or can afford to spend 100% of their time on the bootcamp. So let's say have full time job and do it as a spare time. So there's also alternative to as we know, expensive coding bootcamps.

Great. So the next question is Playwright's a really hot topic nowadays. It's one of the leading main automation tools. How did you learn about Playwright?

Don't remember exactly where I heard about Playwright first time. That was probably like over a year ago, year and a half ago. That was very early stage and the community was just started to grow. And as I mentioned I love to learn like the new technologies right. To try to experiment with it. And that's what my my blog is about my learning path around tools. So then Playwright appear and I started to big like build like the small proof of concepts around it. Yeah.

Awesome. So you've done a lot in a short time frame because a lot of testers I know that been in the industry for a while never even touched performance testing. So how did you get involved with performance then? Like were you working on a project and then you needed something with performance? How did you get into that as well?

That's a really good question. Yeah, what it is. My previous role before Zoom, I was working for the startup in Edinburgh, Paw Print. And in a startup you have this freedom of experimenting with the tools. Like only two QA automation engineers, me and my lead and we could decide what tools we will use. But it also we needed to being happy to be exposed to different testing like performance accessibility, like visual regression tests. So that's how it came about. It was basically oh, can I do it with Playwright? Sure, that you can.

Great. So we're going to get into this. So the next part is, you know someone has Playwright scripts so they want to run an end to end test and then they need to do some sort of performance. Is this what led you to say okay, what kind of tools then can I integrate with Playwright in order for me to get this functionality?

So I guess that there is a few ways you can approach it regarding decision around the tool. Probably speak with your team what they feel comfortable, build a proof of concept what is available. As you know, Playwright gives you this opportunity to access the dev tools and actually check in a browser browser performance. So you can approach it this way as well. Or you can use let's say Artillery as I was using it. But yeah mainly probably speak with the team what they are happy to be exposed because Artillery is a Node JS package so might work fine with your application here. If you are using your testing framework, if you are using Node JS but also introduce YAML syntax. Not everyone might be keen on it, so listen to your team.

So the team say, hey, there's this tool called Artillery that we want to use. Or did you say, oh, I like YAML, so let me like, how do you define Artillery? Because once again, it's not obscure, but it's not one of the more popular performance tools that I'm aware of.

Yes. So learning Artillery was actually really interesting because so, yeah, so we're looking for something that would support Node JS that won't introduce any new language, so the other developers would become comfortable using it and understanding it. So that was the first win for Artillery. And then I think not too long ago I came across this article that I posted my blog post by maintainers of Artillery that the integration with Playwright is possible. So also there was something like, okay, maybe worth trying experiment.

Great, so you knew the tooling then. So you had Playwright, you had Artillery. How did you connect the two then? Like, how did you know? Is there a plugin? Is there? Is it just calling it as a library. How do you get Playwright to I guess talk to Artillery? And the other way around?

Oh yes. So you can use for packages that you would need to run. The test is Artillery, but also Artillery Playwright engine. So that one is sort of plugin processor engine or Artillery that in your YAML file. So in a config of the test, you specify that you would use this engine for your tests, provide the path to processor. So the JavaScript script that can be integrated with Artillery and you basically defined your test cases there this way.

Like I said, I'll actually have a link to this post because you actually have a really good example of the config file itself. So dumb question. What's a processor?

Processor is a JavaScript script. JavaScript file that has methods.

Cool. And so once again, in the link I'll have in the show notes, you show has the processor and then it has some things like the name of the phases and the different durations and everything. How did you come up with the duration or the phases? Is that a template or so that's.

A big so exactly the idea of the so the integration with Playwright and Artillery allows you to use the features of Artillery in your Playwright performance tests. So one of the. Also in my blog post I listed some of those features where I think that they are really, really good and really useful for the tests is like you can define phases of your load of the virtual users, so we can specify the arrival rate and duration of the phase. So let's say the phase would take, I don't know, five minutes, 300 seconds. And with every second there will be five users arriving to completing the journey for the scenario. And that would be, let's say warm up stage with the actually start phase. You can maybe double the number of user or something like this to monitor the performance of the services.

So do you monitor the CPU or the hard drive? What do you usually monitor when you're running these type of performance tests?

Are we talking here about the Artillery or playwright metrics?

I guess both. When you're running playwright with artillery, I guess it must be artillery that does the analysis, right?

Okay, so I will explain. So when you write Artillery, it targets the backend services. So all HTTP requests that you do in your scenario and you measure the response time of the service. So let's say you check it for the one user, you can see it's okay. But when you check it for, as I mentioned, like for 10 users, the response time could be slower. So there's some improvement might be needed here. But when you use it with Playwright, you also add the metrics for the browser. If you ever used let's say Lighthouse extension Browser extension, you right click, you write a lighthouse report performance. You can exactly get those values, similar values for the browser from the playwright artillery. Cool.

So how do you know what kind of performance or what to performance test then? So you had your end to end playwright test. How do you know? Okay, let's make a performance test out of this. And how did you know how many users to run and what the response time should be? Where did you get all that data?

That's really good question. I'm not sure if that's right answer. Actually you may get some data from business analysis from people who would have the metrics related to requirements of the service. We have I don't know, 50 users visiting our application per day, but we are expecting like the load to change 100 will our application handle it? So that's when you would run the performance test to basically make sure that application will handle it or if not when will break. And yeah, this idea of adding performance tests also early in development is really good. Rather than leaving it as a, as a last, you know, oh, we need to make sure that the. Oh, we need to also tick the box for performance testing. That's why I think that if you would add it to your end to end test and execute it along, you can have a fairly good assurance that your application works as expected.

So how often do you run the test with the artillery on? Is it in every check in or do you do it after every sprint? Or like what's the cadence that you would run a performance test against the application?

That's again depends of the team of the resources. Depends also on I guess on the environment use and their limits. You know, like when you use like a lower environment, when maybe there are some sort of limitation, maybe it's better to run it against the higher environments. So I guess there are a few things to consider it.

Cool. So I think you mentioned YAML. Sometimes people don't like YAML, I guess. For the folks that don't know YAML, are there any things you found to make it easier to work with the YAML file when you're running your YAML scenarios?

Yes. So again, here's Artillery, which is amazing in both ways. So I will start from the artillery script scenarios. They may look really complex because to perform complete the user journeys, you execute lots of API calls and then the YAML looks fairly messy. But like in my blog post example and the repo that I provided there, I talk about the supporting tool for Artillery that converts har file that you can get from the network tab in your browser. You can convert it to YAML scenario and it's like so simple and takes seconds that I would really recommend to use it. Also Artillery YAML allows you to add to your tests hooks that if you don't mind, I will mention now as well hooks like before scenario or before request or after scenario and after response. So you can provide here in your processor you can define methods that would be executed exactly as before scenario, before request and so on. I found it really, really useful for running scenarios. Let's say you want to add item to your shopping cart, right? But like the requirements per user needs to be logged in. You don't want to be load testing the path logging in. You know, type email and password and click login. You just want to have authenticated user when you execute the scenario. So here is the before scenario hook and actually in a blog post example, I use actually playwright here. Not playwright artillery, but just a playwright to do the API call to log in the user, you know, the post body email and password to the login endpoint and save the data from the response and use it as a artillery variable for cookie. Let's say that then you can use for before each request to pass it in the header. See, it's just like magic how you can integrate all together, right?

You know, someone listened to like you Said may sound complex. How long did it take you in order to get this right or to get it to work or to understand how it worked? Is this like if you work in JavaScript this is all would be natural to you or is took some time to understand how everything worked together?

I guess it's like lots of time on stack overflow, you know, for people who are looking for similar solutions or maybe I think that the hooks for the YAML scenarios were most complex to investigate because I wanted to do something else there. Because wanted to use playwright there. So I needed to exactly understand how it works. So both documentation for the artillery and the stack overflow.

All right, so because this is audio mainly I just want people to understand, say they have a playwright end to end test already and they want to do a performance test of a. Like I said, an add to cart. So they need an YAML file, they need a library for artillery and then they add like you called. Usually they're called transactions and like loadrunner, but I forgot what you called it. It's like a timer around the functions that you want to keep track of for response times. Can you just walk me through like creating a simple script, what that would look like for someone?

Yeah, of course. So I wouldn't mix two together. Those are different end to end scenario and performance scenario that would be kept in the separate files. Right.

Okay, so that's. That was one of my questions. I didn't know if you added decorator to your playwright script and you could turn them on and off. It sounds like you have to make one a performance test and not reuse it as an end to end.

Yes. So however in the playwright artillery engine playwright you can. You have access to the page. So the same as the playwright you use page click page double click, and so on and so on. You have access to it. However, you don't have access to the test runner. So assertions within the artillery playwright. And you don't have access to all those great features of playwright like setting up context, like making your test unique. And obviously you don't have access to the test runner here. But artillery gives you access to different things. So that would be events and virtual user VU context that would allow you to add some custom metrics to your browser performance tests, change the number of users and so on. You can manipulate use the more artillery related feature within your page script.

So with vusers, I'm just like an old school. A machine had to have so much memory in order to run a vuser and then you had to scale up different machines with all these little agents on it. How does this work with Artillery? Does it scale up per computer or is it. Do you have to kind of have a certain requirement in place for a spec for the machine you're running on?

Yeah, that's a good question because especially with Playwright Artillery you not only spin up the users and exactly exercise the CPU of your machine, but also opens browsers. Right? That's the Playwright part that you have browser. So yeah, there are limits related to machine you execute those tests on and they can like mess up with your metrics because you are not checking actually performance of the service of the browser. You are just checking if your machine handles it. So but Artillery has also pro version, so the paid version when they allow you to run your Artillery tests on the AWS resources. So the scalable. Being honest, I didn't investigate this part. As I said, I just learned for myself with like the fridge here. But I believe there would be like the either EC2 or Fargate services that you can use for to run your performance test on a big scale.

So is there any functionality you didn't cover here that you also you could achieve with Artillery and Playwright?

Maybe I would mention the Artillery feature of using plugins and the one that I was experimenting with was.

Expect. So that allows you to run assertions on your scenarios. So not only just run the scenario and fire the lots of API calls to the backend services, but also adds the exactly assertions through the scenario that you can verify the response status, you can check the response body, you can grab the response object actually and assign the value from it as an environment variable. So it's really flexible tool this way.

So you did mention you were on Stack overflow a lot and I think everyone does that as a developer. But is there anything looking back on that you thought maybe was confusing from the maybe the documentation of Artillery or Playwright that you thought could be clearer that maybe would trip up a lot of people that if they only knew they'd be better off than when you were when you started.

So I don't know. You find the same with documentations for the tool. They're great. Explain like the basic example, you know, like a setup, right. Run init script. Here is how your hello world should look like. But if you want to like do a little bit different then you need to research more and that's absolutely understandable. So I think that the documentation is great. It's just always when you try to do something on a bigger scale, you need to think about implementation as well.

So you said you basically did this on your own. Did developers help at all with maybe like a programmer that really knows JavaScript? Did you ever have to reach out to any of those type of resources to get this running?

Let's start with Edit. If I'll answer wrong here. The thing about being automation engineer, as I think about myself, is that I can do all languages but very basic. So if that would be something that can be written better, more efficient, or towards best practices in the given language, even if it's java, if it's JavaScript, if it's Python, I'm more than happy to listen to advices. Cool.

Okay Kat, before we go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts or playwright artillery efforts? And what's the best way to find or contact you?

So on my blog there's a contact section. There is also link to my LinkedIn contact provides my contact email and the LinkedIn, you know, just a message and.

Do you have any actionable advice to give to someone to help them with their performance testing? In general.

It really depends because I was in a situation that I was new to the performance testing so there was lots of learning. So I would say depends on again on a team if you are a single tester or if there's a team that will support so it's always consideration what tools to use and so on. If the team is already let's say for end to end testing is using using Playwright, it's always better to don't mess up with other tools and just keep Playwright in the artillery. That's my advice.

Thank you Kat for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com a422 and while you're there, make sure to click on the Try for Free Today link under the Exclusive Sponsor section to learn all about Sauce Labs awesome products and services. And if you really want to accelerate your automation testing game in 2023, you want to register for automationguild.com and head on over there now so you can take advantage of the early Bird Special. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is help you succeed at creating end and full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world. We'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real world value that can improve the skills or solve a problem for the Guild community, I love to hear from you. Head ON over to testguild.info and let's make it happen.