Load testing with Artillery and Playwright with Katarzyna Kmiotek

Foreign.

Get ready to discover some of the most actionable performance engineering, testing and site reliability device with some of the world's smartest engineers. Hey, I'm Joe Calantonio, host of the Test Guild and SRE Podcast and my goal is to help you succeed in creating application performance awesomeness.

Hey it's Joe and welcome to another episode of the Test Guild Performance and Site Reliability Podcast. And today in for a special treat, we have the hot topic of Playwright here and Cat is going to be joining us talking all about load testing with Artillery and Playwright. So I'm sure we'll touch on a little bit about functional automation as well, but mostly hopefully how you can actually augment your Playwright scripts with Artillery, which I'm really excited about. If you don't know, Kat is a quality engineer at Zoopla. Enjoys exploring capabilities of Playwright as a functional and non functional test automation tool. So a lot of end to end testing, accessibility testing. She knows everything about visual regression performance, so really excited to have her on the show. I actually came up with this topic by seeing her blog post on this exact topic, which I'll have link for in the show notes so you don't want to miss that. And her most recent post actually explores Artillery AIM playwright integration that allows testing, browser and service performance. So Kat volunteers also as the Codebar Organizer, which is a coding charity for supporting people with underrepresented groups and learning tech. So really excited that she's sharing all her knowledge here as well. So you don't want to miss this episode. Check it out.

This episode is brought to you by the awesome folks at SmartBear. Listen, we know load testing is tough but necessary, so investing in the right tools to automate tests, identify bottlenecks and resolve fast saves your organization both time and money. And that's why SmartBear created Load Ninja, a SaaS load testing tool to help teams get full visibility into performance so you can release quality software faster than ever. Make testing effortless and give it a shot. It's free and easy to try. Head on over to loadninja.com and learn more. Hey Cat, welcome to the Guild.

Pleasure to meet you and happy to be here.

Yeah, great to have you. Really excited about this topic. Like I mentioned, I guess. Before we get into it though, is there anything I missed in your bio that you want the Guild to know more about?

Maybe the fact that I'm a career changer is fairly important for the role that I do as Organizer, co organizer of the Code Bar Chapter, Edinburgh and Glasgow. I try to promote that sort of way of thinking. There's no wrong path. I decided to do career change fairly recent and I cannot imagine my life without tech at the moment. So that's what I try to do in my spare time. Convince as many people as possible, especially female. That is great fun.

Awesome, awesome. So what did you do before then, when you mean career change?

I was working in hospitality roles mainly.

Nice. So how did you get into tech then? That's interesting. So you were in hospitality. How did you even say, oh, I'm going to go into tech all of a sudden?

Yeah, that was big dive. That's true. I had my partner who was supporting me all the way and probably he believed in me more than I did that I can do this transition. So.

Nice.

I try to be that sort of support for other people as well if possible.

All right, so I guess the next question I'm already off topic is there's a lot of things you get into with tech. So how did you fall in then into testing?

So did the coding bootcamp. That was like teaching you as a full stack developer. And the area that I was most interested was like deployment, pipelines, everything that happened behind the scene. And then the opportunity as an automation engineer appeared right after the coding bootcamp. And it was a great experience because I do what I like, being between front and back end somewhere. DevOps.

Great. So this is not scripted. So once again, another question just came to my mind. A lot of people actually asked me about boot camps. I've never been through one, so it sounds like it actually worked for you. So do you recommend that as a valid way for people to maybe make a transition into tech and testing?

Yeah, especially for the people that they are not sure where to start. Because bootcamp provides you with sort of guided path, you know, you learn git and so on, so on in the right order. So that's benefit. But here there is like, let's say Code bar is a charity that supports people who are not that lucky that can afford the boot camp or can afford to spend 100% of their time on the bootcamp. So let's say have full time job and do it as a spare time. So there's also alternative to, as we know, expensive coding bootcamps.

Great. So the next question is, playwright's a really hot topic nowadays. It's one of the leading main automation tools. How did you learn about Playwright?

Don't remember exactly where I heard about Playwright first time. That was probably like over a year ago, year and a half ago. That was very early stage and the community was just started to grow. And as I mentioned, I love to learn like the new technologies. Right. To try to experiment with it. And that's what my, my blog is about of my learning path around tools. So then Playwright appeared and I started to big like build like the small proof of concepts around it.

Yeah, awesome. So you've done a lot in a short time frame because a lot of testers I know that have been in the industry for a while never even touched performance testing. So how did you get involved with performance then? Like were you working on a project and you needed something with performance? How did you get into that as well?

That's a really good question. Yeah, what it is my previous role before I was working for the startup in Edinburgh, Paw Print. And in a startup you have this, you know, freedom of experimenting with the tools. Like only two QA automation engineers, me and my lead and we had like we could decide what tools we'll use. But it also we needed tools being happy to be exposed to different testing like performance accessibility, like visual regression tests. So that's how it came about. It was basically oh, can I do it with Playwright? Sure, that you can.

Great. So we're going to get into this. So the next part is, you know, someone has Playwright scripts so they want to run an end to end test and then they need to do some sort of performance. Is this what led you to say okay, what kind of tools then can I integrate with Playwright in order for me to get this functionality?

So I guess that there is a few ways you can approach it regarding decision around the tool. Probably speak with your team what they feel comfortable, build a proof of concept, what is available. As you know, Playwright gives you this opportunity to access the dev tools and actually check in a browser. Browser performance. So you can approach it this way as well or you can use let's say Artillery as I was using it. But yeah, mainly probably speak with the team what they are happy to be exposed because Artillery is a Node JS package so might work fine with your application here. If you are using your testing framework, if you are using Node JS but also introduce YAML syntax, not everyone might be keen on it. So listen to your team.

So did the team say, hey, there's this tool called Artillery that we want to use or did you say oh, I like YAML. So let me like how did you find Artillery? Because once again it's not obscure, but it's not one of the more popular performance tools I'm aware of.

Yes. So learning Artillery was, was actually really interesting because so yes, so we're looking for something that would support Node JS that won't introduce any, any new language so the other developers would become comfortable using it and understanding it. So that was the first win for Artillery. And then I think not too long ago I came across this article that I posted my blog post by maintainers of Artillery that the integration with Playwright is possible. So also there was something like okay, maybe worth trying experiment.

Great, so you knew the tooling then. So you had Playwright, you had Artillery. How did you connect the two then? Like how did you know? Is there a plugin? Is there, Is it just calling it as a library. How do you get Playwright to I guess talk to Artillery? And the other way around.

Oh yes, so you can use packages that you would need to run the tests is Artillery, but also Artillery Playwright engine. So that one is sort of plugin processor engine for Artillery that in your YAML file. So in a config of the test you specify that you would use this engine for your tests, provide the path to processor. So the JavaScript script that can be integrated with Artillery. And you basically defined your test cases there this way.

Like I said, I'll actually have a link to this post because you actually have a really good example of the config file itself. So dumb question. What's a processor?

Processor is a JavaScript script JavaScript file that has methods.

Cool. And so once again, in the link I'll have in the show notes you show has the processor and then it has some things like the name of the phases and the different durations and everything. How did you come up with the duration or the phases? Is that a template or so that's.

A big so exactly. The idea of the so the integration with Playwright and Artillery allows you to use the features of Artillery in your Playwright performance tests. So one of the also in my blog post I listed some of those features where I think that they are really, really good and really useful for the tests is like you can define phases of your load of the virtual users so we can specify the arrival rate and duration of the phase. So let's say the phase would take, I don't know, five minutes, 300 seconds and with every second there will be five users arriving to completing the journey for the scenario. And that could be, let's say warm up stage with the actually start phase. You can maybe double the number of user or something like this to monitor the performance of the services.

So do you monitor the CPU or the hard drive? What do you usually monitor when you're running these type of performance tests?

Are we talking here about the artillery or playwright metrics?

I guess both. When you're running playwright with artillery, I guess it must be artillery that does the analysis, right?

Okay, so I will explain. So when you write artillery, it targets the backend services. So all HTTP requests that you do in your scenario and you measure the response time of the service. So let's say you check it for the one user, you can see it's okay. But when you check it for as I mentioned, like for 10 users, the response time could be slower. So there is like some improvement might be needed here. But when you use it with Playwright, you also add the metrics for the browser. If you ever used let's say lighthouse extension, browser extension, you right click, you write a lighthouse report performance. You can exactly get those values, similar values for the browser from the playwright artillery. Cool.

So how do you know what kind of performance or what to performance test then? So you had your end to end playwright test. How do you know. Okay, let's make a performance test out of this. And how did you know how many users to run and what the response time should be? Where did you get all that data?

That's really good question. I'm not sure if that's right answer. Actually you may get some data from business analysis from people who would have the metrics related to requirements of the service. We have I don't know, 50 users visiting our application per day, but we are expecting like the load to change 100. Will our application handle it? So that's when you would run the performance test to basically make sure that the application will handle it or if not when will break. And yeah, this idea of adding performance tests also early in development is really good. Rather than leaving it as a, as a last, you know, oh, we need to make sure that the. Oh, we need to also tick the box for performance testing. That's why I think that if you would add it to your end to end test and execute it along, you can have a fairly good assurance that your application works as expected.

So how often do you run the test with the artillery on? Is it in every check in or do you do it after every sprint or like what's the cadence that you would run a performance test against the application?

That's again depends of the team of the resources. Depends also on I guess on the environments use and their limits. You know, like when you use like a lower environment, when maybe there are some sort of limitation. Maybe it's better to run it against the higher environments. So there are a few things to consider.

Cool. So I think you mentioned YAML. Sometimes people don't like YAML, I guess. For the folks that don't know YAML, are there any things you found to make it easier to work with the YAML file when you're running your YAML scenarios?

Yes. So again here's Artillery which is amazing in both ways. So I will start from the artillery script scenarios. They may look really complex because to perform complete the user journeys you execute lots of API calls and then the YAML looks fairly messy. But like in my blog post example and the repo that I provided there, I talk about the supporting tool for Artillery that converts har file that you can get from the network tab in your browser. You can convert it to YAML scenario and it's like so simple and takes seconds that I would really recommend to use it. Also Artillery YAML allows you to add to your tests hooks that if you don't mind I will mention now as well hooks like before scenario or before request or after scenario and after response. So you can provide here in your processor you can define methods that would be executed exactly as before scenario, before request and so on. I found it really really useful for running scenarios. Let's say you want to add item to your shopping cart, right? But like the requirements per user needs to be logged in. You don't want to be load testing the path logging in. You know, type email and password and click login. You just want to have authenticated user when you execute the scenario. So here is the before scenario hook. And actually in a blog post example I use actually playwright here. Not playwright artillery but just a playwright to do the API call to log in the user. You know the post body email and password to the login endpoint and save the data from the response and use it as a artillery variable for cookie. Let's say that then you can use for before each request to pass it in a header. See, it's just like magic how you can integrate all together, right?

You know, so when listen to like you said may sound complex. How long did it take you in order to get this right or to get it to work or to understand how it worked? Is this like if you work in JavaScript this is all would be natural to you or is took some time to understand how everything worked together.

I guess it's like lots of time on stack overflow, you know, for people who are looking for similar solutions. Or maybe I think that the hooks for the YAML scenarios were most complex to investigate because I wanted to do something else there. Because wanted to use playwright there. So I needed to exactly understand how it works. So both documentation for the artillery and the stack overflow.

All right. So because this is audio mainly I just want people to understand, say they have a playwright and end test already and they want to do a performance test of a. Like I said, an add to cart. So they need a YAML file, they need a library for artillery and then they add like called. Usually they're called transactions and like load runner but I forgot what you called it. It's like a timer around the functions that you want to keep track of for response times. Can you just walk me through like creating a simple script, what that would look like for someone?

Yeah, of course. So I wouldn't mix two together. Those are different end to end scenario and performance scenario. They would be kept in the separate files.

Right, okay, so that's. That was one of my questions. I didn't know if you added decorator to your playwright script and you could turn them on and off. It sounds like you have to make one a performance test and not reuse it as an end to end.

Yes. So however, in the playwright artillery engine playwright, you can. You have access to the page. So the same as the playwright, you use page click page double click, and so on and so on. You have access to it. However, you don't have access to the test runner. So assertions within the artillery playwright and you don't have access to all those great features of playwright like setting up context, like making your test unique. And obviously you don't have access to the test runner here. But artillery gives you access to different things. So there would be events and virtual user VU context that would allow you to add some custom metrics to your browser performance tests, change the number of users and so on. You can manipulate use the more artillery related feature within your page script.

So with vusers, I'm just like an old school. A machine had to have so much memory in order to run a vuser and then you had to scale up different machines with all these little agents on it. How does this work with artillery? Does it scale up per computer or is it. Do you have to kind of have a certain requirement in place for a spec for the machine you're running on?

Yeah, that's a good question because especially with playwright artillery, you not only spin up the users and exactly exercise the CPU of your machine, but also opens browsers. Right? That's the playwright Part that you have browser. So yeah there are limits related to machine you execute those tests on and they can like mess up with your metrics because you are not checking actually performance of the service of the browser, you are just checking if your machine handles it. So but Artillery has also pro version so the paid version when they allow you to run your Artillery test on the AWS resources. So the scalable. Being honest, I didn't investigate this part. As I said, I just learned for myself with like the Fruitshare. But I believe there would be like the either EC2 or Fargate services that you can use for to run your performance tests on a big scale.

So is there any functionality you didn't cover here that you could achieve with Artillery and Playwright?

Maybe I would mention the Artillery feature of using plugins and the one that I was experimenting with was.

Expect. So that allows you to run assertions on your scenarios. So not only just run the scenario and fire the lots of API calls to the backend services, but also adds the exactly assertions through the scenario that you can verify the response status, you can check the response body, you can grab the response object actually and assign the value from it as an environment variable. So it's really flexible tool this way.

So you did mention you were on Stack overflow a lot and I think everyone does that as a developer. But is there anything looking back on that you thought maybe was confusing from the maybe the documentation of Artillery or Playwright that you thought could be clearer that maybe would trip up a lot of people that if they only knew they'd be better off than when you were when you started.

So I don't know. You find the same with documentations for the tool. They're great. Explain like the basic example, you know, like a setup, right? Run init script. Here is how your hello world should look like. But if you want to like do a little bit different then you need to research more and that's absolutely understandable. So I think that the documentation is great. It's just always when you try to do something on a bigger scale, you need to think about implementation as well.

So you said you basically did this on your own. Do the developers help at all with maybe like a programmer that really knows JavaScript? Did you ever have to reach out to any of those type of resources to get this running?

Let's start with Edit. If I'll answer wrong here. The thing about being automation engineer, as I think about myself, is that I can do all languages, but very basic. So if there would be something that can be written better, more efficient or towards best practices in the given language. Even if it's java, if it's JavaScript, if it's Python, I'm more than happy to listen to advices. Cool.

Okay Kat, before we go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts or Playwright Artillery efforts? And what's the best way to find or contact you?

So on my blog there's a contact section. There is also link to my LinkedIn contact provides my contact email and the LinkedIn you know, just a message and.

Do you have any actionable advice to give to someone to help them with their performance testing? In general?

It really depends because I was in a situation that I was new to the performance testing so there was lots of learning. So I would say depends on again on a team if you are a single tester or if there's a team that will support so it's always consideration what tools to use and so on. If the team is already let's say for end to end testing is using using Playwright, it's always better to don't mess up with other tools and just keep Playwright in the artillery.

Thanks again for your performance testing awesomeness Links to everything of value we covered in this episode. Head on over to test guild.com P95 and while you're there, make sure to click on the Try Them Bolt Today link under the Exclusive Sponsor section to learn all about SmartBear's two awesome performance test tool solutions, Load Ninja and Load UiPro. And if the show has helped you in any way, why not rate and review it in itunes? Reviews really do matter in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Performance and Site Reliability Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack performance testing awesomeness. As always, test everything and keep the good Cheers. Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps and automation professionals in the world, we'd love to have you join the fam@testguild.com and if you're in the DevOps automation software testing space, or you're a test tool provider and want to offer real world world value that can improve the skills or solve a problem for the Guild community, I'd love to hear from you. Head on over to test skill.info and let's make it happen.