Making Performance Testing Accessible for All with k6 Studio with Mark Meier and Tom Miseur

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability with some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast and my goal is to help you create DevOps toolchain awesomeness. Hey, if you want to learn more about performance testing and making it accessible to everyone on your team, you're in the right place. Because I'm really excited to have Mark and Tom joining us from Grafana Labs to talk all about the new Grafana K6 studio. Really excited about this and also we'll dive into other performance topics as well and why it's important for you, your QA, your devs, and your total DevOps process. You want to miss this episode? Check it out. Hey, before we get into this episode, I want to quickly talk about the silent killer of most DevOps efforts, that is poor user experience. If your app is slow, it's worse than your typical bug. It frustrating and in my experience and many others I talk to on this podcast, frustrated users don't last long. But since slow performance is as sudden, it's hard for standard error monitoring tools to catch. And that's why I really dig SmartBear's Insight Hub. It's an all in one observability solution that offers front end performance monitoring and distributed tracing. Your developers can easily detect and fix and prevent performance bottlenecks before it affects your users. Sounds cool, right? Don't rely anymore on frustrated user feedback, but I always say try it for yourself. Go to smartbeard.com or use our special link down below and try it for free. No credit card required. Hey guys. Welcome to the Guild.

Hey Joe.

Hello.

Great to have you both here. I guess before we get into it, maybe a little background. I know I've covered K6 a few times before, but for the folks that don't know how would you explain what K6 is at a high level?

Yeah, K6 is a open source load and performance testing software originally built to focus on the developer experience. Over time we've evolved that we want to include everyone that's interested in testing and using K6, whether you're developer, SRE in whatever DevOps department or even QA. A performance tester, performance engineer. We're building K6 for everyone.

Love it. All right, so what I love about K6 it really did help developers make it more to shift left performance testing. So I'm curious to know why expand it to make it more accessible. Have you seen Something in the marketplace or what your users are telling you?

Yeah. So K6 began with like developers in mind because we saw shift left was gaining a lot of momentum, but we saw pretty quickly that not every company was shifting left at the same pace. In many orgs, performance testing still lives with QA or performance testing, performance engineering, a platform team, kind of these people that knew what needed to be tested but didn't work directly with the code base. And kind of that realization led us into creating K6 Studio. They're telling us, hey, K6 sounds great, but it's hard to write test scripts. I don't know the system I'm testing specifically.

Love it. So I guess what is Grafana K6 Studio at a high level? Like because you just, I think, released it. Maybe I covered in a new show maybe like a month ago or so. Like maybe it's been around longer, but a little background about it.

Yeah. K6 Studio is a standalone desktop application, Electron app. It is designed to help people record test scripts that they can use and run with K6. So start your recording, we proxy the traffic and we capture the HTTP requests being made. Turn that into a test script for you. You can apply rules and other things to manipulate and transform the test script, but at the end of it, you should be able to easily create a test script for whatever user journey you need to test.

Yeah. Just to say, I think one of the main benefits of K6 Studio is that you're able to define these rules that Mark mentioned that take care of things that you would typically need to do with a K6 script in order to get it to the expected responses from the thing you're testing. So we're really talking about making it easier to reverse engineer HTTP to a large extent. So there's rules for adding verification checks to the script, parameterizing input variables like, you know, credentials or other things that, you know, you would be typing into a web form. And then also the one that's probably the most important is the correlation rule. So correlation is the whole. Your server sending you a dynamic id, you need to make sure you use that dynamic id, so you have to store it into variable. And setting that all up when you're just operating in code can be a bit challenging. So K6 Studio's correlation rule is an attempt to make that easier.

That's. I was going to mention that I started way back, Almost, gosh, over 25 years ago as a performance engineer and correlation was crazy hard. So how does this handle it then? To make it easier?

Yeah. So one of the main things, even before we talk about the rule itself that K6 Studio provides, are purpose built UIs for viewing HTTP data. So if you've ever used K6, the CLI, the open source tool, you'll quickly notice that, you know, there's no way to really inspect the traffic that K6 generates or that it's working with. So you end up having to console log various things like the response body in order to get any kind of information out. So K6 Studio now has the ability to show you both when you create a recording the HTTP request and response data. If you've ever used something like Fiddler, that's a web proxy that is able to show you HTTP traffic, it's very similar to that kind of experience. But then when it comes to the rule itself, you're essentially able to define the parameters of the correlation that you need to perform through the UI instead of having to kind of do that all in code.

Love it, love it. And Tom, I think I saw a demo from you doing a demo and I'll have a link for that down below. But I think you actually show that as well. And it looks very easy for someone that may be a little intimidated at first to try performance testing.

That's right. If you're, you know, familiar with the concepts, I think you'll quickly figure out how things are done in K6 studio. And you know, if you're used to other load and performance testing tools, I think it'll make a lot of sense if you're still completely new to reverse engineering HTTP, you know, you're still going to find it challenging to begin with. But you know, we hope that it's still going to be easier with K6 Studio than, than, you know, just running a script and getting feedback that way.

So a lot of people use JMeter, it's been around forever. They have all a bunch of plugins. They try to make it easy. How would you Compare yourself to JMeter? Is that something you would compare yourself to? Is this something totally different? Do you have a different goal than maybe other, say, other performance tools that are out there?

Yeah, I think like we've long respected JMeter, they've been around a long time. They were like the original load and performance testing tool. They've served a, an audience quite well for forever and a lot of teams are familiar with them. I think where we try to be different than JMETER is we want to bring teams closer together. When I think of JMeter, I view it as, hey, this is a tool designed for a QA type Persona only when I talk to developers, they don't get excited about using JMeter most of the time. The goal with K6 and K6 Studio is, hey, let's bring those teams together so they're not siloed, they're not on two sides of this wall where the testing team is getting this huge backlog of things to test and dev keeps throwing things at them. Let's give them a tool that the company can say you all can use and focus your efforts on maybe lower that backlog. So the testing department isn't under as much pressure. Developers can do some testing, catch issues before it goes to them. So it's kind of all about collaboration from our point of view.

So that's a good point. That's what I loved about K6. It made it more developer friendly. And also you can plug it into your pipeline. So if someone uses the studio, can they still take that script and send it to the developer? And we would have the standard JavaScript. And how does that work so you don't get the separation out? Like this is a GUI tool you need installed on your machine and developers don't understand it.

So two parts on that. Despite our kind of initial thesis hypothesis on this, not all developers want to write code all the time.

Oh really? Okay.

We released K6 Studio and we've gotten feedback from developers that, hey, this is nice. I now don't have to spend all this time learning the K6 scripting API. I can create a workable script in a few minutes rather than having this like blank slate problem in my IDE. So to now answer your question, K6 Studio does produce a JavaScript file. It's creating the JavaScript code that you can kind of run K6 anywhere, whether it's on your local machine, in SCI Pipeline or in Grafana Cloud. K6, it's a working script.

Nice. One other thing, in case people are worried, I think K6 you have an open source version. Is the studio a paid version?

Nope, open source.

Well, what I did see, I think in Tom's demo is what I think helps a pain point is scaling your performance test. Right. It's not really a load test unless you can put a load on and then you have to have these machines set up with the agent, everything going on, orchestrating. Maybe I didn't really. Maybe I didn't follow. I saw a button for Grafana Cloud. Is that something that would help someone? I guess that probably would be a paid service because you have something behind the scenes. Not mandatory though, is that something that if someone wanted to scale up, they have a working store, they could then maybe consume a service that helps them actually scale it up.

Yeah, our point of view on K6 Studio make creating a test script that's going to be open source. We're not going to put that behind a paywall. For our customers that do use Grafana Cloud, K6 for scale, for storage, for collaboration, because they're a Grafana Cloud customer for other products, we're of course going to make that easy for them to go from. Hey, create a case, a test in K6 Studio, send it to Grafana Cloud, start a test right away. I mean, even for a user that's not a customer, I'll just plug that. Grafana Cloud has a free forever tier that's pretty useful on its own and people are welcome to use that. So yeah, we won't. We're not going to pay wall features on creating test scripts, but we will make customers of Grafana Cloud have an easy way to go from Studio to Grafana Cloud.

And also with K6, you know, it was, it's written in go for the most part behind the scenes. And so it's actually very efficient in terms of, you know, the kinds of the number of virtual users you can have running at the same time on a single machine. So even if you weren't using any additional hardware to run a test, you can probably still run a pretty sizable load test with just your own, you know, laptop that you're running K6 from or where you have K6 studio installed.

Nice. Yeah. And that's what I have issue where I had with Loadrunner. You just have all these machines. This was back in the day, now it's probably 10,000 times better. But anyway, another issue I had once again years ago was not necessarily creating the test that was somewhat easier compared to analyzing the results. So is there anything that helps you analyze results? Because that's another thing people tell me, oh yeah, I could use JMeter, but I don't know how to analyze results. I can't see the, the CPU and correlated with the users and all that. It takes something really hard to get up and running. So is that an option or is that something that it helps with as well?

Grafana Cloud? Yes, that's kind of what we build the paid service around. Right. We want to make collaboration, storage of test results, analysis of test results very easy in Grafana Cloud. And because Grafana Cloud itself is an observability platform, we've built integrations into other Grafana Cloud products. So tracing and profiling, for example K6 can kind of initiate and the baggage so that when you run a test, if you're instrumented for tracing or profiling, you can easily find those traces or profiles from your test writing Grafana cloud and correlate things. So no longer maybe hey, my HTTP request is returning with 10 second response time. You can dig in deeper. Hey, is it this? There's particular trace that's interesting or profile that's interesting depending on what you have instrumented.

Nice. So I know a lot of companies are doing like micro architecture with the microservices and APIs. Are there any misconceptions of what developers can do with this tool? I don't know if that makes sense, but maybe they don't necessarily understand what performance testing in or maybe they're missing out of use cases. Is there anything you see like it was interesting how you mentioned how a lot of them say hey, we don't necessarily want to have to code. This makes it a lot easier. Is there any techniques that they may not be aware of that they can get from? Maybe not a load test, but maybe like two users, five users, anything like that?

Yeah, I think just testing in general. Like definitely with tracing and profiling integration we want to encourage a developer like hey, low level, a small test with some nominal amount of virtual users. See what it does to your system in a lower level environment before you promote that code. Maybe you detect, I don't know, there's a function, right? A profile tells you that hey, this function's really inefficient and some developers may game gamify that and say hey, let me make this more efficient, improve it before I promoted to the next step.

Nice. So I also know, I don't know if it was the blog post. Once again, once again I'll have a link for all these down below. But it's a blog post or the video that mentioned this is just the first version and sound like you guys are actually going to invest more in this. So what do you see down the road that's growing into. So people aren't just like oh, I've done players before and no one invests in it. It just dies on on the vine. So are you going to invest in this nipser? Are there any features on the roadmap you could share with us?

Yeah, so initially I won't share specific numbers, but we do see growing installs and after we made the announcement blog post that grew pretty quickly. So very happy to see that gives us the top signal that, hey, we should keep investing in it. I think the thing I'll tease out and get excited about is very soon, depending on when this podcast is posted, we will have support for K6 browser script creation in K6 Studio public preview, so it'll probably have some rough edges and things that we need to adjust, but that's been an ask from a lot of our customers. They like K6 browser, they want to run real browsers. They want to run lots of real browsers in our cloud to do even a browser load test. So we want to make that easier for them.

Yeah, writing this is if you're familiar with the old school technologies, you're probably familiar with Selenium. So you know the whole automating a browser instead of dealing with a lower level HTTP that can sometimes be quite a lot more complex than simply interacting with web elements in the document object model. So so that's what K6 Browser does. We use the same technique Playwright uses to interact with the browser and our API APIs there are very similar as a result as well. But the challenge is of course, you know, writing any kind of automation script from scratch can be very challenging. And so the recorder that we're putting into K6 studio should go a long way to giving you a working script by purely just through interacting with the website as you're recording.

You did mention because it's written in go, it makes it so you can have one machine and really have more. A lot of virtual users. Does that hold true for K6 browser as well? Because if you start opening up browsers I assume obviously going to be more intense, a lot more resources. Are there any rules of thumb on how to not hit a threshold or something or have that affect the performance test?

Yeah, we are unfortunately at the mercy of Google's JavaScript engine v8 engine. It is pretty beefy. And what's worse is that not only is the engine itself going to use a fair amount of CPU and memory, you are at the mercy of how good your web developers were at developing the website that you're testing. We've seen some beasts in our time and it's not always a very well written website. And the trouble there is it's usually the poorly written ones that you want to look at the most, the ones whose performance you want to capture the most. So yeah, if you're dealing with a poorly written website, you will find that K6 when it runs well, K6 itself won't be using a lot of memory and CPU. It's the browser instances that we launch, you know, we're still launching Chrome. It might be headless, but the fact whether or not you see the GUI doesn't actually make much of a difference at all. It still has to do all of the logic of the JavaScript on the page, and that's where you get the high utilization.

Nice. So this sounds like a great solution. Curious to know, a lot of times you did mention some of. A lot of developers don't necessarily want to have to code a performance tester. It may not be familiar with performance testing. So a lot of times testers are more familiar with it. How do you avoid burning down the place? I know a lot of times people get a tool, they'll put like 20,000 transactions in a second and they're like, oh, it broke. They start a fire, they talk to management, they close it down. You're like, it's not realistic. You know, you're crying wolf. How do you avoid issues like that? Do you have any best practices when approaching performance testing?

Especially as a beginner, my biggest piece of advice is start small and don't throw everything at something at once. Over the years, Tom and I laugh about it, but we've often had customers come to us and tell us, I need a million virtual users. I go, you don't. Let's talk about what your actual goals are here. What's the traffic you're getting on your system? So really take that pause, think about what it is that you're testing. Start at a much lower level than you expect and scale up. You'll probably find plenty of issues at levels far below what your goals are, what you think your goals are, or what I don't know. We've had people like, hey, marketing tells us we need a million concurrent users. Like, no, let's take a deep breath and understand if these numbers make sense.

Vendors love that, too. They're like, oh, you need a million. A license for a million virtual users. Sure, you're like that.

I really dislike it. It leads to so much pain and disappointment.

Right? Yeah. Any advice on how to approach that then? If you're marketing saying, a million virtual users. And you're like, I need to educate them in a way where it's not like I'm being condescending, but it's like, we really want to test what needs to be tested.

I think a point of confusion, I think a lot of people have, is they confuse sessions with concurrent users sometimes. And that's like the first thing we sometimes solve and fix. They say, hey, I need a million Virtual users? Well, no, they need to make a million sessions over some period of time. And in the past I've just walked them through the math on that. Right. You need a million sessions over an hour. How long is the average journey that they're going to take? Let's divide that out. Hey, you only need 10,000 concurrent users to reach this many sessions in that period of time or whatever it may be.

Yeah, I find focusing on the actual, you know, assuming we're talking HTTP would be what is your actual HTTP throughput as opposed to number of users? Number of users is still useful to know. You do kind of want to make sure that you are simulating the same number of, you know, concurrent connections, open connections. But in order to gauge how busy something is and you know, what kind of numbers of virtual users you would need, it's best to look at HTTP throughput. You know, requests sent per second, per minute, per day.

Absolutely. And once again, I'm going back to years ago, I used to have to create like, you know, I'd have 5,000 users just browsing, you know, 3,000, you know, with the weights in between buying something. Are you able to schedule different Personas or different users within K6? Does that make sense?

Tom, you want to take it?

Yeah, sure, yeah. We have this concept of scenarios and you can basically think of them as a test definition within a bigger test. So each scenario can have its own number of virtual users ramping profile how long they run for and you can combine and mix and match as many of those as you want. You have full control over that. So yeah, you can totally simulate a handful of users using K6 Browser to actually navigate through a real browser and combine that with web HTTP virtue users that are generating the bulk of the load. Because it's cheaper to do that. You don't necessarily need to run a test of a hundred percent browser users. You can do this hybrid test where you combine the two.

I also know a lot of times people think you do a load test, you're one and done, you ship it. But I know you guys, one of the reasons why K6 is cool is you can integrate it into your CICD pipelines. So, you know, you probably speak to a lot of teams, a lot of different maturity levels. How do you see that normally working running performance in CI cd? Do you have like a separate environment that spins up every quarter or like, is there a cadence usually recommends?

This is a whole lot of. It depends.

And it depends on a lot of factors, more factors than I can probably come up with right now, it's. I think if I were to give a broad answer recommendation. Nightly builds seem to be the best average across everyone. If you could run tests with nightly builds so that when you come in the morning or whenever you start your day, you have clear pass fail criteria about that suite that ran overnight, hey, you ran these tests and these for these, these passed, these didn't. You have a sense for what can be promoted. Move to the next step. What can't your strategy on? If you're spinning up a ephemeral environment just for that test versus hey, you have a dedicated testing environment or your dedicated testing environment is production overnight. That varies greatly and it kind of depends on what that individual organization has kind of set up and designed and prioritized.

I think, you know, it's some companies, their product is critical, that performance is good. And so they take, you know, performance testing a lot more seriously than others who would just run. Maybe, you know, they'll have low testing engagement before a major release of their software. And so they'll, you know, they might set up some CICD jobs at that point. But yeah, it really depends on how important and how prioritized performance testing is for the organization.

Yeah, I think that's a good point to, to raise something with right there. Some companies, they do prioritize, hey, I want to get this into CI. I want to test regularly. And then there's other companies that test maybe once a quarter or once every six months before a huge release. We actually see it's harder for those companies that are testing infrequently. They create so much technical debt and maintenance work of having to rewrite these scripts entirely before that event that becomes really painful for them. Maybe to bring it back to studio. Studio might help them a little bit. But a continuous performance testing practice is always going to kind of pay off in the long run. You detect things sooner. Your test script maintenance is more natural over time. Less debt to worry about.

So I have to ask you, you're probably sick of it. AI. I think AI machine learning is perfect for performance testing. Do you see any practical applications for AI, Especially maybe synthetic data? If someone's doing performance tests, it's always hard to get valid data. Are there any tools outside that maybe Grafana offers that helps with this?

Thinking about the answer I can give here. So yeah, so for AI within K6, I'm sure a lot of your audience, they've probably played around with ChatGPT and GitHub Copilot. If you ask it to write a K6 test script, it does a pretty good job. I've played around with it. I don't like writing things from scratch. I mean, I'm not a developer by, by training. So I use AI to help mock up some basics of a test script. It can hallucinate, it can invent things to import that don't exist. So it's kind of like a nice pair programmer. Within the K6 department, we are looking into how can we leverage AI to help users write test scripts faster. Not 100% sure what that looks like and what that will be right now. Grafana as a whole, we are also looking at AI and kind of how that can live within the Grafana platform. But I'm not the product manager for some of those other departments, so can't speak too deeply about that.

Yeah, I am a huge fan of GitHub Copilot and I use it to help me autocomplete lines of code in script. So I don't use a full blown, you know, having a conversation with an AI and having it basically generate the whole thing. I'm just using it to, you know, line by line give me autocomplete suggestions and I, you know, it's hard to put a number on it, but I feel like it's probably made me 20% faster at creating scripts. Especially once you've already gotten a few scripts or journeys put together for a particular application, you then create a new script, is able to use the content of your other script in the project in order to give you auto completions. And it's pretty incredible at that point how the kinds of things it suggests and the amount of work that's saved as a result. So it's great.

One of my favorite uses of AI is like I'll be talking to a customer and I know like these modules exist in the scripting API and they're trying to do something. I know we can solve that problem. I'll go talk to Copilot or chatgpt and say, hey, these are the modules. This is the example that can you just help me write it real quick in the 30 seconds so I don't need to spend a half hour trying to do it and 95% correct and test it. And here, customer, here's an example. Super helpful.

Love it. Great use cases. Mark, I was looking at the website, I noticed something about an open API for K6 converter. Can you talk a little bit more about that?

Yeah. So the Open API to K6 Converter we had 1 years ago, but it was kind of a Proof of concept that we didn't really invest into in tandem with creating K6 Studio. We actually took a serious look at can we leverage companies that use open API spec swagger files as their source of truth for APIs to help them write K6 test scripts. And that's also available as of recently. It'll take your OpenAPI spec and convert it into a TypeScript client that you can leverage to create K6 test scripts. A lot of developers like this because they're maintaining open API specs. We're also starting to see some some of those QA platform performance engineering teams buy into it because they trust their developers maintain their open API spec. It's their source of truth. So another option with K6 Studio to make test scripting easier.

Okay guys, before we go, is there one piece of actual advice you can give to someone to help them with their K6 performance testing efforts? And what's the best way to find and learn more about K6?

Try out K6 Studio. It's probably what I would say. Yeah, we wrote it for those starting off, you know, Those new to K6 as well as people who are more familiar with it. But it's going to be easier to do this, what you're trying to do when you have a bit of a user interface to work with, giving you information, relevant information. So definitely try out K6 Studio in Grafana Cloud K6. We do have some test authoring functionality as well that would help people new to load and performance testing as well.

So yeah, I think my biggest piece of advice is start small and iterate. Don't make mountains out of molehills. You'll you can get a lot of information just by testing small things. If you're a developer, you probably know what the most important APIs are for your service or services. Test those critical ones. If you're in performance testing qa, one of these kind of centralized teams, you probably know the critical user journeys for your product, your company. You can leverage K6 Studio to record those journeys and test those. Don't try to test it all at once. And certainly don't try to throw the most amount of virtual users at the system at once. Scale up, start with a small test, prove it works there iterate. I always thought testing was it's just an implementation of the scientific method and we're just going to run in that circle constantly.

All right, before we wrap it up, remember frustrated users quit apps. Don't rely on bad App store reviews. Use SmartBear's Insight Hub to catch fix and prevent performance bottlenecks and crashes from affecting your users. Go to smartbear.com or use the link down below and try for free for 14 days. No credit card required. And for links of everything of value we covered in this DevOps Toolchain show, head on over to test guild.com p184 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation, software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.